Process and Apparatus to Reduce Declared Capacity of a Storage Device by Moving Data

ABSTRACT

Systems, methods and/or devices are used to reduce declared capacity of non-volatile memory of a storage device in a storage system. In one aspect, the method includes, detecting an amelioration trigger for reducing declared capacity of non-volatile memory of a storage device, and in accordance with the detected amelioration trigger, performing an amelioration process to reduce declared capacity of the non-volatile memory of the storage device, the performing including: moving a portion of data used by a host from the storage device to another storage device of the storage system, and reducing declared capacity of the non-volatile memory of the storage device. In some embodiments, the storage device includes one or more flash memory devices. In some embodiments, the detecting, the performing, or both are performed by the storage device, or by one or more subsystems of the storage system distinct from the storage device, or by the host.

RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationSer. No. 62/044,976, filed Sep. 2, 2014, which is herein incorporated byreference in its entirety.

This application is related to the following applications, each of whichis herein incorporated by reference in its entirety:

U.S. patent application Ser. No. ______, filed ______, attorney docket058752-01-5254-US, which claims priority to U.S. Provisional PatentApplication Ser. No. 62/044,883, filed Sep. 2, 2014;

U.S. patent application Ser. No. ______, filed ______, attorney docket058752-01-5255-US, which claims priority to U.S. Provisional PatentApplication Ser. No. 62/044,919, filed Sep. 2, 2014;

U.S. patent application Ser. No. ______, filed ______, attorney docket058752-01-5256-US, which claims priority to U.S. Provisional PatentApplication Ser. No. 62/044,890, filed Sep. 2, 2014;

U.S. patent application Ser. No. ______, filed ______, attorney docket058752-01-5257-US, which claims priority to U.S. Provisional PatentApplication Ser. No. 62/044,898, filed Sep. 2, 2014;

U.S. patent application Ser. No. ______, filed ______, attorney docket058752-01-5258-US, which claims priority to U.S. Provisional PatentApplication Ser. No. 62/044,905, filed Sep. 2, 2014;

U.S. patent application Ser. No. ______, filed ______, attorney docket058752-01-5259-US, which claims priority to U.S. Provisional PatentApplication Ser. No. 62/044,981, filed Sep. 2, 2014;

U.S. patent application Ser. No. ______, filed ______, attorney docket058752-01-5260-US, which claims priority to U.S. Provisional PatentApplication Ser. No. 62/044,989, filed Sep. 2, 2014;

U.S. patent application Ser. No. ______, filed ______, attorney docket058752-01-5261-US, which claims priority to U.S. Provisional PatentApplication Ser. No. 62/044,983, filed Sep. 2, 2014;

U.S. patent application Ser. No. ______, filed ______, attorney docket058752-01-5262-US, which claims priority to U.S. Provisional PatentApplication Ser. No. 62/044,963, filed Sep. 2, 2014;

U.S. patent application Ser. No. ______, filed ______, attorney docket058752-01-5263-US, which claims priority to U.S. Provisional PatentApplication Ser. No. 62/044,930, filed Sep. 2, 2014;

U.S. patent application Ser. No. ______, filed ______, attorney docket058752-01-5264-US, which claims priority to U.S. Provisional PatentApplication Ser. No. 62/044,969, filed Sep. 2, 2014;

U.S. patent application Ser. No. ______, filed ______, attorney docket058752-01-5266-US, which claims priority to U.S. Provisional PatentApplication Ser. No. 62/044,932, filed Sep. 2, 2014; and

U.S. patent application Ser. No. ______, filed ______, attorney docket058752-01-5267-US, which claims priority to U.S. Provisional PatentApplication Ser. No. 62/044,936, filed Sep. 2, 2014.

TECHNICAL FIELD

The disclosed embodiments relate generally to memory systems, and inparticular, to reducing declared capacity of a storage device (e.g.,comprising one or more flash memory devices).

BACKGROUND

Semiconductor memory devices, including flash memory, typically utilizememory cells to store data as an electrical value, such as an electricalcharge or voltage. A flash memory cell, for example, includes a singletransistor with a floating gate that is used to store a chargerepresentative of a data value. Flash memory is a non-volatile datastorage device that can be electrically erased and reprogrammed. Moregenerally, non-volatile memory (e.g., flash memory, as well as othertypes of non-volatile memory implemented using any of a variety oftechnologies) retains stored information even when not powered, asopposed to volatile memory, which requires power to maintain the storedinformation. Increases in storage density have been facilitated invarious ways, including increasing the density of memory cells on a chipenabled by manufacturing developments, and transitioning fromsingle-level flash memory cells to multi-level flash memory cells, sothat two or more bits can be stored by each flash memory cell.

Repeated erasure and reprogramming of flash memory cells causesdegradation of the charge storage capability (wear). Eventually, thecharge storage capability degrades to the point where it becomesimpossible or infeasible to recover the original data (e.g., anunrecoverable codeword is read from the flash memory device, thecomputational resources required to recover a codeword exceed apredefined threshold, or a count of program-erase (PE) cycles for theflash memory device exceeds a threshold value) and the device isconsidered worn out. Wear-leveling technology is often used todistribute the wear across the multiple portions of a flash memorydevice. In a typical system, once the wear limit of a portion of a flashmemory device is reached, the entire flash memory device is consideredto have failed.

SUMMARY

Various embodiments of systems, methods and devices within the scope ofthe appended claims each have several aspects, no single one of which issolely responsible for the attributes described herein. Without limitingthe scope of the appended claims, after considering this disclosure, andparticularly after considering the section entitled “DetailedDescription” one will understand how the aspects of various embodimentsare used to enable reducing declared capacity of a storage device. Inone aspect, an amelioration trigger for reducing declared capacity ofnon-volatile memory of a storage device in a storage system is detected,and in accordance with the detected amelioration trigger, anamelioration process to reduce declared capacity of the non-volatilememory of the storage device is performed.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the present disclosure can be understood in greater detail, amore particular description may be had by reference to the features ofvarious embodiments, some of which are illustrated in the appendeddrawings. The appended drawings, however, merely illustrate pertinentfeatures of the present disclosure and are therefore not to beconsidered limiting, for the description may admit to other effectivefeatures.

FIG. 1A is a block diagram illustrating an implementation of a datastorage system, in accordance with some embodiments.

FIG. 1B is a block diagram illustrating an implementation of a datastorage system, in accordance with some embodiments.

FIG. 1C is a block diagram illustrating an implementation of a datastorage system, in accordance with some embodiments.

FIG. 2A-1 is a block diagram illustrating an implementation of amanagement module, in accordance with some embodiments.

FIG. 2A-2 is a block diagram illustrating an implementation of amanagement module, in accordance with some embodiments.

FIG. 2B-1 is a block diagram illustrating an implementation of a systemmanagement module, in accordance with some embodiments.

FIG. 2B-2 is a block diagram illustrating an implementation of a systemmanagement module, in accordance with some embodiments.

FIG. 2C-1 is a block diagram illustrating an implementation of a clustermanagement module, in accordance with some embodiments.

FIG. 2C-2 is a block diagram illustrating an implementation of a clustermanagement module, in accordance with some embodiments.

FIG. 2D is a block diagram illustrating an implementation of anamelioration module included in FIGS. 2A-1 and 2A-2, in accordance withsome embodiments.

FIG. 3 is a block diagram of a logical address space, and morespecifically a logical block address (LBA) space, in accordance withsome embodiments.

FIG. 4 is a block diagram of a mapping table and physical address space,in accordance with some embodiments.

FIG. 5A is a prophetic diagram of voltage distributions that may befound in a single-level flash memory cell (SLC) over time, in accordancewith some embodiments.

FIG. 5B is a prophetic diagram of voltage distributions that may befound in a multi-level flash memory cell (MLC) over time, in accordancewith some embodiments.

FIG. 6 illustrates a flowchart representation of a method of managing astorage system, in accordance with some embodiments.

FIGS. 7A-7D illustrate a flowchart representation of a method ofmanaging a storage system, in accordance with some embodiments.

FIGS. 8A-8C illustrate a flowchart representation of a method ofmanaging a storage system, in accordance with some embodiments.

FIGS. 9A-9C illustrate a flowchart representation of a method ofmanaging a storage system, in accordance with some embodiments.

In accordance with common practice the various features illustrated inthe drawings may not be drawn to scale. Accordingly, the dimensions ofthe various features may be arbitrarily expanded or reduced for clarity.In addition, some of the drawings may not depict all of the componentsof a given system, method or device. Finally, like reference numeralsmay be used to denote like features throughout the specification andfigures.

DETAILED DESCRIPTION

When a multi-level flash cell has reached its wear limit it typicallystill has charge retention capability sufficient to store a reducednumber of charge levels. Often it is the case that a substantial numberof erasure and reprogramming cycles can be performed on a wear-limitedmulti-level flash cell with full recovery of the stored data, providedthat a reduced number of charge levels is used and expected. Forexample, a flash memory device operating in 3 bits per cell mode (TLC)typically can perform between 500 and 1500 erasure and reprogrammingcycles before being considered worn out. However, at that point in timeit will typically still have sufficient charge storage capability tooperate in the single bit per cell mode (SLC) for an additional 10,000to 20,000 erasure and reprogramming cycles before the SLC wear limit isencountered. Thus the lifetime of the flash memory device may beextended provided that it can be allowed to store less data. Currentlyit is difficult for a storage system to utilize this extended capabilitybecause storage system mechanisms for managing and working with astorage device whose capacity decreases with usage, by decreasingover-provisioning, are inadequate. Consequently what is desired aremechanisms for managing and working with such a storage device,including mechanisms to inform the surrounding system of its impending(or imminent) reduction in capacity so that the system may adjust itsoperation accordingly. Potentially, memory devices with other forms ofnon-volatile memory may benefit from the same or similar mechanisms asthose described in this document.

The various embodiments described herein include systems, methods and/ordevices used to reduce declared capacity of a storage device inaccordance with a detected amelioration trigger. Some embodimentsinclude systems, methods and/or devices to detect a trigger condition inaccordance with one or more metrics of a storage device and enable anamelioration process associated with the detected trigger condition, theamelioration process to reduce declared capacity of non-volatile memoryof the storage device.

(I1) More specifically, some embodiments include a method of managing astorage system. In some embodiments, the method includes: (1) detectingan amelioration trigger for reducing declared capacity of non-volatilememory of a storage device of the storage system, and (2) in accordancewith the detected amelioration trigger, performing an ameliorationprocess to reduce declared capacity of the non-volatile memory of thestorage device, the performing including altering an encoding format ofat least a portion of the non-volatile memory of the storage device, andreducing declared capacity of the non-volatile memory of the storagedevice.

(I1-1) In some embodiments of the method of I1, the method furtherincludes: (1) prior to detecting the amelioration trigger, detecting afirst wear condition of the non-volatile memory of the storage device,wherein a total storage capacity of the non-volatile memory of thestorage device includes declared capacity and over-provisioning, and (2)in response to detecting the first wear condition, performing a remedialaction that reduces over-provisioning of the non-volatile memory of thestorage device without reducing declared capacity of the non-volatilememory of the storage device.

(I1-2) In some embodiments of the method of I1-1, detecting theamelioration trigger includes detecting a second wear condition distinctfrom the first wear condition.

(I2) In some embodiments of the method of I1 or I1-1 or I1-2, thedetecting, the performing, or both the detecting and the performing areperformed by the storage device

(I3) In some embodiments of the method of I1 or I1-1 or I1-2, thedetecting, the performing, or both the detecting and the performing areperformed by one or more subsystems of the storage system distinct fromthe storage device.

(I4) In some embodiments of the method of I1 or I1-1 or I1-2, thedetecting, the performing, or both the detecting and the performing areperformed by the host.

(I5) In some embodiments of the method of I4, the host includes a clienton behalf of which data is stored in the storage system.

(I6) In some embodiments of the method of I4, the host includes astorage system controller of the storage system.

(I7) In some embodiments of the method of I4, the host includes acluster controller of the storage system.

(I8) In some embodiments of the method of any of I1 to I7, altering theencoding format of at least a portion of the non-volatile memory of thestorage device includes altering the encoding format of at least aportion of the non-volatile memory of the storage device in accordancewith one or more parameters for the amelioration process.

(I9) In some embodiments of the method of any of I1 to I8, altering theencoding format of at least a portion of the non-volatile memory of thestorage device includes altering the encoding format from ahigher-density physical encoding format to a lower-density physicalencoding format.

(I10) In some embodiments of the method of any of I1 to I9, altering theencoding format of at least a portion of the non-volatile memory of thestorage device includes altering a number of states per memory cell froma higher number of states to a lower number of states.

(I11) In some embodiments of the method of any of I1 to I10, alteringthe encoding format of at least a portion of the non-volatile memory ofthe storage device includes altering the encoding format from aTriple-Level Cell (TLC) format to a Multi-Level Cell (MLC) format.

(I12) In some embodiments of the method of any of I1 to I11, alteringthe encoding format of at least a portion of the non-volatile memory ofthe storage device includes altering the encoding format from aTriple-Level Cell (TLC) format to a Single-Level Cell (SLC) format.

(I13) In some embodiments of the method of any of I1 to I12, alteringthe encoding format of at least a portion of the non-volatile memory ofthe storage device includes altering the encoding format from aMulti-Level Cell (MLC) format to a Single-Level Cell (SLC) format.

(I14) In some embodiments of the method of any of I1 to I13, performingthe amelioration process to reduce declared capacity of the non-volatilememory of the storage device includes reducing utilization of thenon-volatile memory of the storage device.

(I15) In some embodiments of the method of any of I1 to I14, alteringthe encoding format of at least a portion of the non-volatile memory ofthe storage device includes altering the encoding format of a memoryportion of a plurality of memory portions of the non-volatile memory ofthe storage device.

(I16) In some embodiments of the method of any of I1 to I15, alteringthe encoding format of at least a portion of the non-volatile memory ofthe storage device includes altering the encoding format of all clientdata of the non-volatile memory of the storage device.

(I17) In some embodiments of the method of any of I1 to I16, performingthe amelioration process to reduce declared capacity of the non-volatilememory of the storage device further includes advertising a reduceddeclared capacity of the non-volatile memory of the storage device.

(I18) In some embodiments of the method of any of I1 to I17, the methodfurther comprises (1) after beginning performance of the ameliorationprocess to reduce declared capacity of the non-volatile memory of thestorage device, detecting an indication to abort the reduction indeclared capacity of the non-volatile memory of the storage device, and(2) in response to detecting the indication to abort the reduction indeclared capacity of the non-volatile memory of the storage device,aborting performance of the amelioration process to reduce declaredcapacity of the non-volatile memory of the storage device.

(I19) In some embodiments of the method of any of I1 to I18, the storagedevice comprises one or more flash memory devices.

(I20) In another aspect, a storage device includes (1) non-volatilememory, (2) one or more processors, and (3) controller memory storingone or more programs, which when executed by the one or more processorscause the storage device to perform or control performance of any of themethods I1-I19 described herein.

(I22) In yet another aspect, any of the methods I1-I19 described aboveare performed by a storage device including means for performing any ofthe methods described herein.

(I24) In yet another aspect, a storage system includes (1) a storagemedium (e.g., comprising one or more non-volatile storage devices, suchas flash memory devices) (2) one or more processors, and (3) memorystoring one or more programs, which when executed by the one or moreprocessors cause the storage system to perform or control performance ofany of the methods I1-I19 described herein.

(I25) In yet another aspect, some embodiments include a non-transitorycomputer readable storage medium, storing one or more programsconfigured for execution by one or more processors of a host system, theone or more programs including instructions for performing any of themethods I1-I19 described herein.

(I26) In yet another aspect, a storage system includes (1) one or morestorage devices, (2) one or more subsystems having one or moreprocessors, and (3) memory storing one or more programs, which whenexecuted by the one or more processors cause the one or more subsystemsto perform or control performance of any of the methods I1-I19 describedherein.

(I28) In yet another aspect, a host system includes (1) an interface foroperatively coupling to a storage system, (2) one or more processors,and (3) controller memory storing one or more programs, which whenexecuted by the one or more processors cause the host system to performor control performance of any of the methods I1-I19 described herein.

(K1) More specifically, some embodiments include a method of managing astorage system. In some embodiments, the method includes: (1) detectingan amelioration trigger for reducing declared capacity of non-volatilememory of a storage device of the storage system, and (2) in accordancewith the detected amelioration trigger, performing an ameliorationprocess to reduce declared capacity of the non-volatile memory of thestorage device, the performing including deleting from the storagedevice discardable data that is used by a host, and reducing declaredcapacity of the non-volatile memory of the storage device.

(K1-1) In some embodiments of the method of K1, the method furtherincludes: (1) prior to detecting the amelioration trigger, detecting afirst wear condition of the non-volatile memory of the storage device,wherein a total storage capacity of the non-volatile memory of thestorage device includes declared capacity and over-provisioning, and (2)in response to detecting the first wear condition, performing a remedialaction that reduces over-provisioning of the non-volatile memory of thestorage device without reducing declared capacity of the non-volatilememory of the storage device.

(K1-2) In some embodiments of the method of K1-1, detecting theamelioration trigger includes detecting a second wear condition distinctfrom the first wear condition.

(K2) In some embodiments of the method of K1 or K1-1 or K1-2, the hostincludes a client on behalf of which data is stored in the storagesystem.

(K3) In some embodiments of the method of K1 or K1-1 or K1-2, the hostincludes a storage system controller of the storage system.

(K4) In some embodiments of the method of K1 or K1-1 or K1-2, the hostincludes a cluster controller of the storage system.

(K5) In some embodiments of the method of any of K1 to K4, thedetecting, the performing, or both the detecting and the performing areperformed by the storage device

(K6) In some embodiments of the method of any of K1 to K4, thedetecting, the performing, or both the detecting and the performing areperformed by one or more subsystems of the storage system distinct fromthe storage device.

(K7) In some embodiments of the method of any of K1 to K4, thedetecting, the performing, or both the detecting and the performing areperformed by the host.

(K8) In some embodiments of the method of any of K1 to K7, deletingdiscardable data that is used by the host includes deleting discardabledata that is used by the host in accordance with one or more parametersfor the amelioration process.

(K9) In some embodiments of the method of any of K1 to K8, deletingdiscardable data that is used by the host includes deleting temporarydata.

(K10) In some embodiments of the method of any of K1 to K9, deletingdiscardable data that is used by the host includes deleting data thatcan be rebuilt when needed.

(K11) In some embodiments of the method of any of K1 to K10, deletingdiscardable data that is used by the host includes deleting snapshots ofdata.

(K12) In some embodiments of the method of any of K1 to K11, deletingdiscardable data that is used by the host includes deleting data that ispre-marked by the host as data that is discardable.

(K13) In some embodiments of the method of any of K1 to K12, deletingdiscardable data that is used by the host includes: invalidating one ormore logical address entries, associated with the discardable data, of amapping table, the mapping table used to translate logical addresses ina logical address space to physical addresses in a physical addressspace of the storage device.

(K14) In some embodiments of the method of any of K13, reducing declaredcapacity of the non-volatile memory of the storage device includesmaking a number of logical addresses, less than or equal to a number oflogical addresses corresponding to the invalidated logical addressentries, unavailable to the host.

(K15) In some embodiments of the method of any of K1 to K14, performingthe amelioration process to reduce declared capacity of the non-volatilememory of the storage device further includes advertising a reduceddeclared capacity of the non-volatile memory of the storage device.

(K16) In some embodiments of the method of any of K1 to K15, the methodfurther comprises (1) after beginning performance of the ameliorationprocess to reduce declared capacity of the non-volatile memory of thestorage device, detecting an indication to abort the reduction indeclared capacity of the non-volatile memory of the storage device, and(2) in response to detecting the indication to abort the reduction indeclared capacity of the non-volatile memory of the storage device,aborting performance of the amelioration process to reduce declaredcapacity of the non-volatile memory of the storage device.

(K17) In some embodiments of the method of any of K1 to K16, the storagedevice comprises one or more flash memory devices.

(K18) In another aspect, a storage device includes (1) non-volatilememory, (2) one or more processors, and (3) controller memory storingone or more programs, which when executed by the one or more processorscause the storage device to perform or control performance of any of themethods K1-K17 described herein.

(K20) In yet another aspect, any of the methods K1-K17 described aboveare performed by a storage device including means for performing any ofthe methods described herein.

(K22) In yet another aspect, a storage system includes (1) a storagemedium (e.g., comprising one or more non-volatile storage devices, suchas flash memory devices) (2) one or more processors, and (3) memorystoring one or more programs, which when executed by the one or moreprocessors cause the storage system to perform or control performance ofany of the methods K1-K17 described herein.

(K23) In yet another aspect, some embodiments include a non-transitorycomputer readable storage medium, storing one or more programsconfigured for execution by one or more processors of a storage device,the one or more programs including instructions for performing any ofthe methods K1-K17 described herein.

(K24) In yet another aspect, a storage system includes (1) one or morestorage devices, (2) one or more subsystems having one or moreprocessors, and (3) memory storing one or more programs, which whenexecuted by the one or more processors cause the one or more subsystemsto perform or control performance of any of the methods K1-K17 describedherein.

(K26) In yet another aspect, a host system includes (1) an interface foroperatively coupling to a storage system, (2) one or more processors,and (3) controller memory storing one or more programs, which whenexecuted by the one or more processors cause the host system to performor control performance of any of the methods K1-K17 described herein.

In yet another aspect, some embodiments include a non-transitorycomputer readable storage medium, storing one or more programsconfigured for execution by one or more processors of a host system, theone or more programs including instructions for performing any of themethods described herein.

(L1) More specifically, some embodiments include a method of managing astorage system. In some embodiments, the method includes: (1) detectingan amelioration trigger for reducing declared capacity of non-volatilememory of a first storage device of the storage system, and (2) inaccordance with the detected amelioration trigger, performing anamelioration process to reduce declared capacity of the non-volatilememory of the first storage device, the performing including moving aportion of data that is used by a host from the first storage device toanother storage device of the storage system, and reducing declaredcapacity of the non-volatile memory of the first storage device.

(L1-1) In some embodiments of the method of L1, the method furtherincludes: (1) prior to detecting the amelioration trigger, detecting afirst wear condition of the non-volatile memory of the first storagedevice, wherein a total storage capacity of the non-volatile memory ofthe first storage device includes declared capacity andover-provisioning, and (2) in response to detecting the first wearcondition, performing a remedial action that reduces over-provisioningof the non-volatile memory of the first storage device without reducingdeclared capacity of the non-volatile memory of the first storagedevice.

(L1-2) In some embodiments of the method of L1-1, detecting theamelioration trigger includes detecting a second wear condition distinctfrom the first wear condition.

(L2) In some embodiments of the method of L1 or L1-1 or L1-2, the hostincludes a client on behalf of which data is stored in the storagesystem.

(L3) In some embodiments of the method of L1 or L1-1 or L1-2, the hostincludes a storage system controller of the storage system.

(L4) In some embodiments of the method of L1 or L1-1 or L1-2, the hostincludes a cluster controller of the storage system.

(L5) In some embodiments of the method of any of L1 to L4, thedetecting, the performing, or both the detecting and the performing areperformed by the host.

(L6) In some embodiments of the method of any of L1 to L4, thedetecting, the performing, or both the detecting and the performing areperformed by one or more subsystems of the storage system distinct fromthe first storage device.

(L7) In some embodiments of the method of any of L1 to L6, moving theportion of data that is used by the host includes moving the portion ofdata that is used by the host in accordance with one or more parametersfor the amelioration process.

(L8) In some embodiments of the method of any of L1 to L7, moving theportion of data that is used by the host includes moving data associatedwith one or more virtual logical addresses, including updating a virtualaddress mapping module.

(L9) In some embodiments of the method of any of L1 to L8, moving theportion of the data includes selecting one or more logical addresses ofthat data so as to minimize performance degradation.

(L10) In some embodiments of the method of any of L1 to L8, moving theportion of the data includes selecting one or more logical addresses ofthat data so as to minimize overhead from garbage collection.

(L11) In some embodiments of the method of any of L8 to L10, performingthe amelioration process to reduce declared capacity of the non-volatilememory of the first storage device further includes: invalidating one ormore logical address entries, associated with the portion of data, of amapping table, the mapping table used to translate logical addresses inthe logical address space to physical addresses in a physical addressspace of the first storage device.

(L12) In some embodiments of the method of L11, reducing declaredcapacity of the non-volatile memory of the first storage device includestrimming logical addresses associated with the portion of data movedfrom the first storage device.

(L13) In some embodiments of the method of any of L1 to L12, performingthe amelioration process to reduce declared capacity of the non-volatilememory of the first storage device further includes advertising areduced declared capacity of the non-volatile memory of the firststorage device.

(L14) In some embodiments of the method of any of L1 to L13, the methodfurther comprises: (1) after beginning performance of the ameliorationprocess to reduce declared capacity of the non-volatile memory of thefirst storage device, detecting an indication to abort the reduction indeclared capacity of the non-volatile memory of the first storagedevice; and (2) in response to detecting the indication to abort thereduction in declared capacity of the non-volatile memory of the firststorage device, aborting performance of the amelioration process toreduce declared capacity of the non-volatile memory of the first storagedevice.

(L15) In some embodiments of the method of any of L1 to L14, the firststorage device comprises one or more flash memory devices.

(L16) In another aspect, a host system includes: (1) an interface foroperatively coupling to a storage system, (2) one or more processors,and (3) controller memory storing one or more programs, which whenexecuted by the one or more processors cause the first storage device toperform or control performance of any of the methods L1-L15 describedherein.

(L18) In yet another aspect, any of the methods L1-L15 described aboveare performed by a system including means for performing any of themethods described herein.

(L20) In yet another aspect, a storage system includes (1) a storagemedium (e.g., comprising one or more non-volatile storage devices, suchas flash memory devices) (2) one or more processors, and (3) memorystoring one or more programs, which when executed by the one or moreprocessors cause the storage system to perform or control performance ofany of the methods L1-L15 described herein.

(L21) In yet another aspect, some embodiments include a non-transitorycomputer readable storage medium, storing one or more programsconfigured for execution by one or more processors of a storage device,the one or more programs including instructions for performing any ofthe methods L1-L15 described herein.

(L22) In yet another aspect, a storage system includes (1) one or morestorage devices, (2) one or more subsystems having one or moreprocessors, and (3) memory storing one or more programs, which whenexecuted by the one or more processors cause the one or more subsystemsto perform or control performance of any of the methods L1-L15 describedherein.

Numerous details are described herein in order to provide a thoroughunderstanding of the example embodiments illustrated in the accompanyingdrawings. However, some embodiments may be practiced without many of thespecific details, and the scope of the claims is only limited by thosefeatures and aspects specifically recited in the claims. Furthermore,well-known methods, components, and circuits have not been described inexhaustive detail so as not to unnecessarily obscure pertinent aspectsof the embodiments described herein.

Data storage systems, including those described below, use a variety oftechniques to avoid data loss caused by a variety of failure mechanisms,including storage media failure, communication failures, and failures atthe system and subsystem level. A common feature of these mechanisms isthe use of data redundancy to protect data, to compensate for actual andpotential data errors (e.g., media errors, lost data, transmissionerrors, inaccessible data, etc.). One class of redundancy mechanisms isknown as error correction codes (ECCs). Numerous types of errorcorrection codes are well known (e.g., BCH, LDPC, Reed-Solomon, etc.),as are numerous schemes for storing them with or in conjunction with thedata that is being protected. Another class of redundancy mechanisms iserasure codes (e.g., pyramid, fountain, partial MDS, locally repairable,simple regenerating, etc.)

Another type or level of redundancy mechanism is typically called RAID(redundant array of independent disks), even when the storage media arenot “disks” in the traditional sense. There are multiple forms of RAID,or RAID schemes, providing different levels of data protection (e.g.,RAID-1, RAID-5, RAID-6, RAID-10, etc.). Typically, in systems that useRAID, “stripes” of data stored in multiple distinct storage locationsare treated as a set, and stored with sufficient redundant data that anydata in a stripe that would have been lost, in a partial or completefailure of any one of the storage locations, is recovered using theother data in the stripe, possibly including the redundant data.

A third type of redundancy mechanism is replication of data to multiplestorage locations, typically in distinct failure domains. Systemsimplementing this type of redundancy mechanism typically store three ormore replicas of each data set or data item. Typically either eachreplica is in a distinct failure domain from the other replicas, or atleast one replica is in a distinct failure domain from the otherreplicas.

The embodiments described below work in conjunction with the dataredundancy mechanisms described above (used alone or in combination).Some of the data storage systems described below have an architecture orconfiguration designed to implement a particular redundancy mechanism.Furthermore, some of the embodiments described below may utilize morethan one of the redundancy mechanisms described above, either alone orin combination. Furthermore, some of the embodiments are able to storedata encoded with different redundancy mechanisms simultaneously.Furthermore, even within a single mechanism, the selection of parameters(i.e., codeword size relative to data size) may vary dynamically. Hence,altering the redundancy mechanism directly affects the amount of datastored and in turn the utilization.

FIG. 1A is a block diagram illustrating data storage system 100, inaccordance with some embodiments. While some example features areillustrated, various other features have not been illustrated for thesake of brevity and so as not to obscure pertinent aspects of theexample embodiments disclosed herein. To that end, as a non-limitingexample, data storage system 100 includes a storage device 120, whichincludes a storage controller 124 and a storage medium 130, and is usedin conjunction with or includes a computer system 110. In someembodiments, storage medium 130 is a single flash memory device while inother embodiments storage medium 130 includes a plurality of flashmemory devices. In some embodiments, storage medium 130 is NAND-typeflash memory or NOR-type flash memory. In some embodiments, storagemedium 130 includes one or more three-dimensional (3D) memory devices,as further defined herein. Further, in some embodiments storagecontroller 124 is a solid-state drive (SSD) controller. However, othertypes of storage media may be included in accordance with aspects of awide variety of embodiments (e.g., PCRAM, ReRAM, STT-RAM, etc.). In someembodiments, a flash memory device includes one or more flash memorydie, one or more flash memory packages, one or more flash memorychannels or the like. In some embodiments, data storage system 100 cancontain one or more storage device 120 s.

Computer system 110 is coupled to storage controller 124 through dataconnections 101. However, in some embodiments computer system 110includes storage controller 124, or a portion of storage controller 124,as a component and/or a subsystem. For example, in some embodiments,some or all of the functionality of storage controller 124 isimplemented by software executed on computer system 110. Computer system110 may be any suitable computer device, such as a computer, a laptopcomputer, a tablet device, a netbook, an internet kiosk, a personaldigital assistant, a mobile phone, a smart phone, a gaming device, acomputer server, or any other computing device. Computer system 110 issometimes called a host, host system, client, or client system. In someembodiments, computer system 110 is a server system, such as a serversystem in a data center. In some embodiments, computer system 110includes one or more processors, one or more types of memory, a displayand/or other user interface components such as a keyboard, a touchscreen display, a mouse, a track-pad, a digital camera and/or any numberof supplemental devices to add functionality. In some embodiments,computer system 110 does not have a display and other user interfacecomponents.

Storage medium 130 is coupled to storage controller 124 throughconnections 103. Connections 103 are sometimes called data connections,but typically convey commands in addition to data, and optionally conveymetadata, error correction information and/or other information inaddition to data values to be stored in storage medium 130 and datavalues read from storage medium 130. In some embodiments, however,storage controller 124 and storage medium 130 are included in the samedevice (i.e., an integral device) as components thereof. Furthermore, insome embodiments, storage controller 124 and storage medium 130 areembedded in a host device (e.g., computer system 110), such as a mobiledevice, tablet, other computer or computer controlled device, and themethods described herein are performed, at least in part, by theembedded memory controller. Storage medium 130 may include any number(i.e., one or more) of memory devices including, without limitation,non-volatile semiconductor memory devices, such as flash memorydevice(s). For example, flash memory device(s) can be configured forenterprise storage suitable for applications such as cloud computing,for database applications, primary and/or secondary storage, or forcaching data stored (or to be stored) in secondary storage, such as harddisk drives. Additionally and/or alternatively, flash memory device(s)can also be configured for relatively smaller-scale applications such aspersonal flash drives or hard-disk replacements for personal, laptop,and tablet computers. In some embodiments, storage medium 130 includesone or more three-dimensional (3D) memory devices, as further definedherein.

Storage medium 130 is divided into a number of addressable andindividually selectable blocks, such as selectable portion 131. In someembodiments, the individually selectable blocks are the minimum sizeerasable units in a flash memory device. In other words, each blockcontains the minimum number of memory cells that can be erasedsimultaneously. Each block is usually further divided into a pluralityof pages and/or word lines, where each page or word line is typically aninstance of the smallest individually accessible (readable) portion in ablock. In some embodiments (e.g., using some types of flash memory), thesmallest individually accessible unit of a data set, however, is asector, which is a subunit of a page. That is, a block includes aplurality of pages, each page contains a plurality of sectors, and eachsector is the minimum unit of data for reading data from the flashmemory device.

As noted above, while data storage densities of non-volatilesemiconductor memory devices are generally increasing, a drawback ofincreasing storage density is that the stored data is more prone tobeing stored and/or read erroneously. In some embodiments, error controlcoding can be utilized to limit the number of uncorrectable errors thatare introduced by electrical fluctuations, defects in the storagemedium, operating conditions, device history, write-read circuitry,etc., or a combination of these and various other factors.

In some embodiments, storage controller 124 includes a management module121-1, a host interface 129, a storage medium I/O interface 128, andadditional module(s) 125. Storage controller 124 may include variousadditional features that have not been illustrated for the sake ofbrevity and so as not to obscure pertinent features of the exampleembodiments disclosed herein, and a different arrangement of featuresmay be possible. Host interface 129 provides an interface to computersystem 110 through data connections 101. Similarly, storage medium I/O128 provides an interface to storage medium 130 though connections 103.In some embodiments, storage medium I/O 128 includes read and writecircuitry, including circuitry capable of providing reading signals tostorage medium 130 (e.g., reading threshold voltages for NAND-type flashmemory).

In some embodiments, management module 121-1 includes one or moreprocessing units (CPUs, also sometimes called processors) 122-1configured to execute instructions in one or more programs (e.g., inmanagement module 121-1). In some embodiments, the one or more CPUs122-1 are shared by one or more components within, and in some cases,beyond the function of storage controller 124. Management module 121-1is coupled to host interface 129, additional module(s) 125 and storagemedium I/O 128 in order to coordinate the operation of these components.In some embodiments, one or more modules of management module 121-1 areimplemented in management module 121-2 of computer system 110. In someembodiments, one or more processors of computer system 110 (not shown)are configured to execute instructions in one or more programs (e.g., inmanagement module 121-2). Management module 121-2 is coupled to storagedevice 120 in order to manage the operation of storage device 120.

Additional module(s) 125 are coupled to storage medium I/O 128, hostinterface 129, and management module 121-1. As an example, additionalmodule(s) 125 may include an error control module to limit the number ofuncorrectable errors inadvertently introduced into data during writes tomemory or reads from memory. In some embodiments, additional module(s)125 are executed in software by the one or more CPUs 122-1 of managementmodule 121-1, and, in other embodiments, additional module(s) 125 areimplemented in whole or in part using special purpose circuitry (e.g.,to perform encoding and decoding functions). In some embodiments,additional module(s) 125 are implemented in whole or in part by softwareexecuted on computer system 110.

In some embodiments, an error control module, included in additionalmodule(s) 125, includes an encoder and a decoder. In some embodiments,the encoder encodes data by applying an error control code to produce acodeword, which is subsequently stored in storage medium 130. When theencoded data (e.g., one or more codewords) is read from storage medium130, the decoder applies a decoding process to the encoded data torecover the data, and to correct errors in the recovered data within theerror correcting capability of the error control code. Those skilled inthe art will appreciate that various error control codes have differenterror detection and correction capacities, and that particular codes areselected for various applications for reasons beyond the scope of thisdisclosure. As such, an exhaustive review of the various types of errorcontrol codes is not provided herein. Moreover, those skilled in the artwill appreciate that each type or family of error control codes may haveencoding and decoding algorithms that are particular to the type orfamily of error control codes. On the other hand, some algorithms may beutilized at least to some extent in the decoding of a number ofdifferent types or families of error control codes. As such, for thesake of brevity, an exhaustive description of the various types ofencoding and decoding algorithms generally available and known to thoseskilled in the art is not provided herein.

In some embodiments, during a write operation, host interface 129receives data to be stored in storage medium 130 from computer system110. The data received by host interface 129 is made available to anencoder (e.g., in additional module(s) 125), which encodes the data toproduce one or more codewords. The one or more codewords are madeavailable to storage medium I/O 128, which transfers the one or morecodewords to storage medium 130 in a manner dependent on the type ofstorage medium being utilized.

In some embodiments, a read operation is initiated when computer system(host) 110 sends one or more host read commands (e.g., via dataconnections 101, or alternatively a separate control line or bus) tostorage controller 124 requesting data from storage medium 130. Storagecontroller 124 sends one or more read access commands to storage medium130, via storage medium I/O 128, to obtain raw read data in accordancewith memory locations (addresses) specified by the one or more host readcommands. Storage medium I/O 128 provides the raw read data (e.g.,comprising one or more codewords) to a decoder (e.g., in additionalmodule(s) 125). If the decoding is successful, the decoded data isprovided to host interface 129, where the decoded data is made availableto computer system 110. In some embodiments, if the decoding is notsuccessful, storage controller 124 may resort to a number of remedialactions or provide an indication of an irresolvable error condition.

As explained above, a storage medium (e.g., storage medium 130) isdivided into a number of addressable and individually selectable blocksand each block is optionally (but typically) further divided into aplurality of pages and/or word lines and/or sectors. While erasure of astorage medium is performed on a block basis, in many embodiments,reading and programming of the storage medium is performed on a smallersubunit of a block (e.g., on a page basis, word line basis, or sectorbasis). In some embodiments, the smaller subunit of a block consists ofmultiple memory cells (e.g., single-level cells or multi-level cells).In some embodiments, programming is performed on an entire page. In someembodiments, a multi-level cell (MLC) NAND flash typically has fourpossible states per cell, yielding two bits of information per cell.Further, in some embodiments, a MLC NAND has two page types: (1) a lowerpage (sometimes called fast page), and (2) an upper page (sometimescalled slow page). In some embodiments, a triple-level cell (TLC) NANDflash has eight possible states per cell, yielding three bits ofinformation per cell. Although the description herein uses TLC, MLC, andSLC as examples, those skilled in the art will appreciate that theembodiments described herein may be extended to memory cells that havemore than eight possible states per cell, yielding more than three bitsof information per cell.

The encoding format of the storage media (i.e., TLC, MLC, or SLC and/ora chose data redundancy mechanism) is a choice made when data isactually written to the storage media. Often in this specification thereis described an event, condition, or process that is said to set theencoding format, alter the encoding format of the storage media, etc. Itshould be recognized that the actual process may involve multiple steps,e.g., erasure of the previous contents of the storage media followed bythe data being written using the new encoding format and that theseoperations may be separated in time from the initiating event, conditionor procedure.

As an example, if data is written to a storage medium in pages, but thestorage medium is erased in blocks, pages in the storage medium maycontain invalid (e.g., stale) data, but those pages cannot beoverwritten until the whole block containing those pages is erased. Inorder to write to the pages with invalid data, the pages (if any) withvalid data in that block are read and re-written to a new block and theold block is erased (or put on a queue for erasing). This process iscalled garbage collection. After garbage collection, the new blockcontains the pages with valid data and may have free pages that areavailable for new data to be written, and the old block can be erased soas to be available for new data to be written. Since flash memory canonly be programmed and erased a limited number of times, the efficiencyof the algorithm used to pick the next block(s) to re-write and erasehas a significant impact on the lifetime and reliability of flash-basedstorage systems.

FIG. 1B is a block diagram illustrating data storage system 140, inaccordance with some embodiments. While some example features areillustrated, various other features have not been illustrated for thesake of brevity and so as not to obscure pertinent aspects of theexample embodiments disclosed herein. To that end, as a non-limitingexample, data storage system 140 (sometimes called a scale-up storagesystem, a single node storage system, etc.) includes a plurality ofstorage devices 160 (e.g., storage devices 160-1 to 160-m) and a storagesystem controller 150, and is used in conjunction with a computer system142. In some embodiments, storage devices 160 include management modules161 (e.g., storage device 160-1 includes management module 161-1 andstorage device 160-m includes management module 161-m). Some of thefeatures described above with respect to storage device 120 (FIG. 1A)and management module 121-1 (FIG. 1A) are applicable to storage devices160 and management modules 161, respectively, and for sake of brevityand simplicity, the details are not repeated here.

Computer system 142 is coupled to storage system controller 150 throughconnections 141. However, in some embodiments computer system 142includes a part of or the entire storage system controller 150 as acomponent and/or a subsystem. For example, in some embodiments, some orall of the functionality of storage system controller 150 is implementedby software executed on computer system 142. Computer system 142 may beany suitable computer device, such as a computer, a laptop computer, atablet device, a netbook, an internet kiosk, a personal digitalassistant, a mobile phone, a smart phone, a gaming device, a computerserver, or any other computing device. In some embodiments, computersystem 142 is a server system, such as a server system in a data center.Computer system 142 is sometimes called a host, host system, client, orclient system. In some embodiments, computer system 142 includes one ormore processors, one or more types of memory, a display and/or otheruser interface components such as a keyboard, a touch screen display, amouse, a track-pad, a digital camera and/or any number of supplementaldevices to add functionality. In some embodiments, computer system 142does not have a display and other user interface components.

In some embodiments, storage system controller 150 includes a systemmanagement module 151-1, and additional module(s) 155. Storage systemcontroller 150 may include various additional features that have notbeen illustrated for the sake of brevity and so as not to obscurepertinent features of the example embodiments disclosed herein, and adifferent arrangement of features may be possible. For example, in someembodiments, storage system controller 150 additionally includes aninterface for each of the storage devices 160 coupled to storage systemcontroller 150. Storage devices 160 are coupled to storage systemcontroller 150 through connections 143 (e.g., storage device 160-1 iscoupled to storage system controller 150 through connections 143-1 andstorage device 160-m is coupled to storage system controller 150 throughconnections 143-m). In some embodiments, connections 143-1 through 143-mare implemented as a communication media over which commands and dataare communicated using a protocol such as SCSI, SATA, Infiniband,Ethernet, Token Ring, or the like.

In some embodiments, system management module 151-1 includes one or moreprocessing units (CPUs, also sometimes called processors) 152-1configured to execute instructions in one or more programs (e.g., insystem management module 151-1). In some embodiments, the one or moreCPUs 152-1 are shared by one or more components within, and in somecases, beyond the function of storage system controller 150. Systemmanagement module 151-1 is coupled to additional module(s) 155 in orderto coordinate the operation of these components. In some embodiments,one or more modules of system management module 151-1 are implemented insystem management module 151-2 of computer system 142 (sometimes calleda host, host system, client, or client system). In some embodiments, oneor more processors (sometimes called CPUs or processing units) ofcomputer system 142 (not shown) are configured to execute instructionsin one or more programs (e.g., in system management module 151-2).System management module 151-2 is coupled to storage system controller150 in order to manage the operation of storage system controller 150.

Additional module(s) 155 are coupled to system management module 151-1.In some embodiments, additional module(s) 155 are executed in softwareby the one or more CPUs 152-1 of system management module 151-1, and, inother embodiments, additional module(s) 155 are implemented in whole orin part using special purpose circuitry. In some embodiments, additionalmodule(s) 155 are implemented in whole or in part by software executedon computer system 142.

During a write operation, storage system controller 150 receives data tobe stored in storage devices 160 from computer system 142 (sometimescalled a host, host system, client, or client system). In someembodiments, storage system controller 150 maps a virtual logicaladdress from computer system 142 to an address, which determines oridentifies the one or more of storage devices 160 to which to write thedata.

A read operation is initiated when computer system 142 sends one or morehost read commands to storage system controller 150 requesting data fromstorage devices 160. In some embodiments, storage system controller 150maps a virtual logical address from computer system 142 to an address,which determines or identifies the one or more of storage devices 160from which to read the requested data.

FIG. 1C is a block diagram illustrating an implementation of datastorage system 170, in accordance with some embodiments. While someexample features are illustrated, various other features have not beenillustrated for the sake of brevity and so as not to obscure pertinentaspects of the example embodiments disclosed herein. To that end, as anon-limiting example, data storage system 170 (sometimes called ascale-out storage system, a multiple node storage system or a storagecluster system) includes a plurality of storage subsystems 192 (e.g.,storage subsystems 192-1 to 192-s) and a cluster controller 180, and isused in conjunction with a computer system 172. In some embodiments,storage subsystems 192 include storage system controllers 190 andstorage devices 194 (e.g., storage subsystem 192-1 includes storagesystem controller 190-1 and storage devices 194-1 through 194-n). Someof the features described above with respect to data storage system 140(FIG. 1B) are applicable to storage subsystems 192, and for sake ofbrevity, the details are not repeated here. In some embodiments theremay be a plurality of cluster controllers 180 that may communicate witheach other to coordinate their activities.

Computer system 172 is coupled to cluster controller 180 throughconnections 171. However, in some embodiments computer system 172includes cluster controller 180 as a component and/or a subsystem. Forexample, in some embodiments, some or all of the functionality ofcluster controller 180 is implemented by software executed on computersystem 172. Computer system 172 may be any suitable computer device,such as a computer, a laptop computer, a tablet device, a netbook, aninternet kiosk, a personal digital assistant, a mobile phone, a smartphone, a gaming device, a computer server, or any other computingdevice. In some embodiments, computer system 172 is a server system,such as a server system in a data center. Computer system 172 issometimes called a host, host system, client, or client system. In someembodiments, computer system 172 includes one or more processors, one ormore types of memory, a display and/or other user interface componentssuch as a keyboard, a touch screen display, a mouse, a track-pad, adigital camera and/or any number of supplemental devices to addfunctionality. In some embodiments, computer system 172 does not have adisplay and other user interface components.

In some embodiments, cluster controller 180 includes a clustermanagement module 181-1, and additional module(s) 185. Clustercontroller 180 may include various additional features that have notbeen illustrated for the sake of brevity and so as not to obscurepertinent features of the example embodiments disclosed herein, and adifferent arrangement of features may be possible. For example, in someembodiments, cluster controller 180 additionally includes an interfacefor each of the storage subsystems 192 coupled to cluster controller180. Storage subsystems 192 are coupled to cluster controller 180through connections 173 (e.g., storage subsystem 192-1 is coupled tocluster controller 180 through connections 173-1 and storage subsystem192-s is coupled to cluster controller 180 through connections 173-s).In some embodiments, connections 173 may be implemented as a sharedcommunication network, e.g., Token Ring, Ethernet, Infiniband, etc.

In some embodiments, cluster management module 181-1 includes one ormore processing units (CPUs, also sometimes called processors) 182-1configured to execute instructions in one or more programs (e.g., incluster management module 181-1). In some embodiments, the one or moreCPUs 182-1 are shared by one or more components within, and in somecases, beyond the function of cluster controller 180. Cluster managementmodule 181-1 is coupled to additional module(s) 185 in order tocoordinate the operation of these components. In some embodiments, oneor more modules of cluster management module 181-1 are implemented incluster management module 181-2 of computer system 172 (sometimes calleda host, host system, client, or client system). In some embodiments, oneor more processors (sometimes called CPUs or processing units) ofcomputer system 172 (not shown) are configured to execute instructionsin one or more programs (e.g., in cluster management module 181-2).Cluster management module 181-2 is coupled to cluster controller 180 inorder to manage the operation of cluster controller 180.

Additional module(s) 185 are coupled to cluster management module 181-1.In some embodiments, additional module(s) 185 are executed in softwareby the one or more CPUs 182-1 of cluster management module 181-1, and,in other embodiments, additional module(s) 185 are implemented in wholeor in part using special purpose circuitry. In some embodiments,additional module(s) 185 are implemented in whole or in part by softwareexecuted on computer system 172.

In some embodiments, during a write operation, cluster controller 180receives data to be stored in storage subsystems 192 from computersystem 172 (sometimes called a host, host system, client, or clientsystem). In some embodiments, cluster controller 180 maps a virtuallogical address from computer system 172 to an address formatunderstandable by storage subsystems 192 and to identify a storagesubsystem of storage subsystems 192 to which to write the data. In someembodiments, cluster controller 180 may convert the data to be storedinto a plurality of sets of data, each set of data is stored on onestorage subsystem of storage subsystems 192. In one embodiment, theconversion process may be as simple as a partitioning of the data to bestored. In another embodiment, the conversion process may redundantlyencode the data to be stored so as to provide enhanced data integrityand access in the face of failures of one or more storage subsystems ofstorage subsystems 192 or communication thereto.

In some embodiments, a read operation is initiated when computer system172 sends one or more host read commands to cluster controller 180requesting data from storage subsystems 192. In some embodiments,cluster controller 180 maps a virtual logical address from computersystem 172 to an address format understandable by storage subsystems192, to determine or identify the storage subsystem of storagesubsystems 192 from which to read the requested data. In someembodiments, more than one storage subsystem of storage subsystems 192may have data read in order to satisfy the read operation, e.g. for datareconstruction.

As used herein, the term “host” or “host system” may be construed tomean (1) a computer system (e.g., computer system 110, FIG. 1A, computersystem 142, FIG. 1B, or computer system 172, FIG. 1C) on behalf of whichdata is stored in a storage system (e.g., data storage system 100, FIG.1A, data storage system 140, FIG. 1B, or data storage system 170, FIG.1C), (2) a storage system controller (e.g., storage system controller150, FIG. 1B) of a storage system (e.g., data storage system 140, FIG.1B), (3) a cluster controller (e.g., cluster controller 180, FIG. 1C) ofa storage system (e.g., data storage system 170, FIG. 1C), and/or (4)any computing entity (e.g., a computer, a process running on a computer,a mobile phone, an internet kiosk, a tablet computer, a laptop computer,a desktop computer, a server computer, etc.) that is operatively coupledeither directly or indirectly to a storage system, depending on thecontext. For example, in some circumstances, with respect to datastorage system 140 (FIG. 1B), the term “host” may refer to computersystem 142 or storage system controller 150, depending on the context.As another example, in some circumstances, with respect to data storagesystem 170 (FIG. 1C), the term “host” may refer to computer system 172or cluster controller 180, depending on the context. Further, in somecontexts, the host is or includes a client or client system, on behalfof which data is stored in a storage system.

FIG. 2A-1 is a block diagram illustrating a management module 121-1, inaccordance with some embodiments, as shown in FIG. 1A. Management module121-1 typically includes one or more processing units (sometimes calledCPUs or processors) 122-1 for executing modules, programs and/orinstructions stored in memory 206-1 and thereby performing processingoperations, memory 206-1 (sometimes called controller memory), and oneor more communication buses 208-1 for interconnecting these components.The one or more communication buses 208-1 optionally include circuitry(sometimes called a chipset) that interconnects and controlscommunications between system components. Management module 121-1 iscoupled to host interface 129, additional module(s) 125, and storagemedium I/O 128 by the one or more communication buses 208-1. Memory206-1 includes high-speed random access memory, such as DRAM, SRAM, DDRRAM or other random access solid state memory devices, and may includenon-volatile memory, such as one or more magnetic disk storage devices,optical disk storage devices, flash memory devices, or othernon-volatile solid state storage devices. Memory 206-1 optionallyincludes one or more storage devices remotely located from the CPU(s)122-1. Memory 206-1, or alternatively the non-volatile memory device(s)within memory 206-1, comprises a non-transitory computer readablestorage medium. In some embodiments, memory 206-1, or the non-transitorycomputer readable storage medium of memory 206-1 stores the followingprograms, modules, and data structures, or a subset or superset thereof:

-   -   translation table 212-1 that is used for mapping logical        addresses to physical addresses (e.g., in some embodiments,        translation table 212-1 includes mapping table 402, FIG. 4);    -   data read module 214-1 that is used for reading data from one or        more codewords, pages or blocks in a storage medium (e.g.,        storage medium 130, FIG. 1A);    -   data write module 216-1 that is used for writing data to one or        more codewords, pages or blocks in a storage medium (e.g.,        storage medium 130, FIG. 1A);    -   data erase module 218-1 that is used for erasing data from one        or more blocks in a storage medium (e.g., storage medium 130,        FIG. 1A);    -   garbage collection module 220-1 that is used for garbage        collection for one or more blocks in a storage medium (e.g.,        storage medium 130, FIG. 1A);    -   metrics module 222-1 that is used for generating and/or        obtaining one or more metrics of a storage device (e.g., storage        device 120, FIG. 1A);    -   trigger detection module 224-1 that is used for detecting a        trigger condition (e.g., in accordance with one or more metrics        of a storage device);    -   enabling module 226-1 that is used for enabling an amelioration        process associated with a trigger condition (e.g., detected by        trigger detection module 224-1);    -   notification module 228-1 that is used for notifying a host to        which a storage device is operatively coupled of a trigger        condition (e.g., detected by trigger detection module 224-1)        and/or of an absence of the trigger condition;    -   amelioration module 230-1 that is used for performing an        amelioration process to reduce declared capacity of non-volatile        memory of a storage device (e.g., storage device 120, FIG. 1A),        optionally including:        -   detection module 231-1 that is used for detecting an            amelioration trigger for reducing declared capacity of the            non-volatile memory of the storage device;        -   utilization module 232-1 that is used for reducing            utilization of the non-volatile memory of the storage            device; and        -   capacity module 234-1 that is used for reducing declared            capacity of the non-volatile memory of the storage device.

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and corresponds to a set ofinstructions for performing a function described above. The aboveidentified modules or programs (i.e., sets of instructions) need not beimplemented as separate software programs, procedures or modules, andthus various subsets of these modules may be combined or otherwisere-arranged in various embodiments. In some embodiments, memory 206-1may store a subset of the modules and data structures identified above.Furthermore, memory 206-1 may store additional modules and datastructures not described above. In some embodiments, the programs,modules, and data structures stored in memory 206-1, or thenon-transitory computer readable storage medium of memory 206-1, provideinstructions for implementing some of the methods described below. Insome embodiments, some or all of these modules may be implemented withspecialized hardware circuits that subsume part or all of the modulefunctionality.

Although FIG. 2A-1 shows management module 121-1 in accordance with someembodiments, FIG. 2A-1 is intended more as a functional description ofthe various features which may be present in management module 121-1than as a structural schematic of the embodiments described herein. Inpractice, and as recognized by those of ordinary skill in the art, theprograms, modules, and data structures shown separately could becombined and some programs, modules, and data structures could beseparated.

FIG. 2A-2 is a block diagram illustrating a management module 121-2 ofcomputer system 110 (FIG. 1A), in accordance with some embodiments.Management module 121-2 typically includes one or more processing units(sometimes called CPUs or processors) 122-2 for executing modules,programs and/or instructions stored in memory 206-2 and therebyperforming processing operations, memory 206-2, and one or morecommunication buses 208-2 for interconnecting these components. The oneor more communication buses 208-2 optionally include circuitry(sometimes called a chipset) that interconnects and controlscommunications between system components. Management module 121-2 iscoupled to storage device 120 by the one or more communication buses208-2. Memory 206-2 (sometimes called host memory) includes high-speedrandom access memory, such as DRAM, SRAM, DDR RAM or other random accesssolid state memory devices, and may include non-volatile memory, such asone or more magnetic disk storage devices, optical disk storage devices,flash memory devices, or other non-volatile solid state storage devices.Memory 206-2 optionally includes one or more storage devices remotelylocated from the CPU(s) 122-2. Memory 206-2, or alternatively thenon-volatile memory device(s) within memory 206-2, comprises anon-transitory computer readable storage medium. In some embodiments,memory 206-2, or the non-transitory computer readable storage medium ofmemory 206-2 stores the following programs, modules, and datastructures, or a subset or superset thereof:

-   -   translation table 212-2 that is used for mapping logical        addresses to physical addresses (e.g., in some embodiments,        translation table 212-2 includes mapping table 402, FIG. 4);    -   data read module 214-2 that is used for reading data from one or        more codewords, pages or blocks in a storage medium (e.g.,        storage medium 130, FIG. 1A);    -   data write module 216-2 that is used for writing data to one or        more codewords, pages or blocks in a storage medium (e.g.,        storage medium 130, FIG. 1A);    -   data erase module 218-2 that is used for erasing data from one        or more blocks in a storage medium (e.g., storage medium 130,        FIG. 1A);    -   garbage collection module 220-2 that is used for garbage        collection for one or more blocks in a storage medium (e.g.,        storage medium 130, FIG. 1A);    -   metrics module 222-2 that is used for generating and/or        obtaining one or more metrics of a storage device (e.g., storage        device 120, FIG. 1A);    -   trigger detection module 224-2 that is used for detecting a        trigger condition (e.g., in accordance with one or more metrics        of a storage device);    -   enabling module 226-2 that is used for enabling an amelioration        process associated with a trigger condition (e.g., detected by        trigger detection module 224-2);    -   notification module 228-2 that is used for notifying an        application, module or process, of the host (i.e., computer        system 110, FIG. 1A), of a trigger condition (e.g., detected by        trigger detection module 224-2) and/or of an absence of the        trigger condition;    -   amelioration module 230-2 that is used for performing an        amelioration process to reduce declared capacity of non-volatile        memory of a storage device (e.g., storage device 120, FIG. 1A),        optionally including:        -   detection module 231-2 that is used for detecting an            amelioration trigger for reducing declared capacity of the            non-volatile memory of the storage device;        -   utilization module 232-2 that is used for reducing            utilization of the non-volatile memory of the storage            device; and        -   capacity module 234-2 that is used for reducing declared            capacity of the non-volatile memory of the storage device.

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and corresponds to a set ofinstructions for performing a function described above. The aboveidentified modules or programs (i.e., sets of instructions) need not beimplemented as separate software programs, procedures or modules, andthus various subsets of these modules may be combined or otherwisere-arranged in various embodiments. In some embodiments, memory 206-2may store a subset of the modules and data structures identified above.Furthermore, memory 206-2 may store additional modules and datastructures not described above. In some embodiments, the programs,modules, and data structures stored in memory 206-2, or thenon-transitory computer readable storage medium of memory 206-2, provideinstructions for implementing some of the methods described below. Insome embodiments, some or all of these modules may be implemented withspecialized hardware circuits that subsume part or all of the modulefunctionality.

Although FIG. 2A-2 shows management module 121-2 in accordance with someembodiments, FIG. 2A-2 is intended more as a functional description ofthe various features which may be present in management module 121-2than as a structural schematic of the embodiments described herein. Inpractice, and as recognized by those of ordinary skill in the art, theprograms, modules, and data structures shown separately could becombined and some programs, modules, and data structures could beseparated.

FIG. 2B-1 is a block diagram illustrating a system management module151-1, in accordance with some embodiments, e.g., embodiments in whichthe system management module is in a storage system controller, as shownin FIG. 1B. System management module 151-1 typically includes one ormore processing units (sometimes called CPUs or processors) 152-1 forexecuting modules, programs and/or instructions stored in memory 246-1and thereby performing processing operations, memory 246-1 (sometimescalled storage system controller memory or controller memory), and oneor more communication buses 248-1 for interconnecting these components.The one or more communication buses 248-1 optionally include circuitry(sometimes called a chipset) that interconnects and controlscommunications between system components. System management module 151-1is coupled to additional module(s) 155 by the one or more communicationbuses 248-1. Memory 246-1 includes high-speed random access memory, suchas DRAM, SRAM, DDR RAM or other random access solid state memorydevices, and may include non-volatile memory, such as one or moremagnetic disk storage devices, optical disk storage devices, flashmemory devices, or other non-volatile solid state storage devices.Memory 246-1 optionally includes one or more storage devices remotelylocated from the CPU(s) 152-1. Memory 246-1, or alternatively thenon-volatile memory device(s) within memory 246-1, comprises anon-transitory computer readable storage medium. In some embodiments,memory 246-1, or the non-transitory computer readable storage medium ofmemory 246-1 stores the following programs, modules, and datastructures, or a subset or superset thereof:

-   -   system mapping module 250-1 that is used for mapping virtual        logical addresses (e.g., used by computer system 142, FIG. 1B)        to intermediate addresses (e.g., which are mapped by storage        devices 160 to physical addresses, FIG. 1B);    -   metrics module 252-1 that is used for generating and/or        obtaining one or more metrics of a storage device (e.g., any of        storage devices 160-1 to 160-m, FIG. 1B);    -   trigger detection module 254-1 that is used for detecting a        trigger condition (e.g., in accordance with one or more metrics        of a storage device);    -   enabling module 256-1 that is used for enabling an amelioration        process associated with a trigger condition (e.g., detected by        trigger detection module 254-1);    -   notification module 258-1 that is used for notifying a host to        which a storage device is operatively coupled of a trigger        condition (e.g., detected by trigger detection module 254-1)        and/or of an absence of the trigger condition;    -   amelioration module 260-1 that is used for performing an        amelioration process to reduce declared capacity of non-volatile        memory of a storage device (e.g., storage device 160, FIG. 1B),        optionally including:        -   detection module 261-1 that is used for detecting an            amelioration trigger for reducing declared capacity of the            non-volatile memory of the storage device;        -   utilization module 262-1 that is used for reducing            utilization of the non-volatile memory of the storage            device; and        -   capacity module 264-1 that is used for reducing declared            capacity of the non-volatile memory of the storage device;            and    -   optionally, data redundancy module 266-1 that is used for        redundantly encoding data (e.g., to implement a particular RAID        (redundant array of independent disks) level); and    -   optionally, communication module 268-1 that is used for        facilitating communications with other devices, for example via        a storage area network (SAN).

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and corresponds to a set ofinstructions for performing a function described above. The aboveidentified modules or programs (i.e., sets of instructions) need not beimplemented as separate software programs, procedures or modules, andthus various subsets of these modules may be combined or otherwisere-arranged in various embodiments. In some embodiments, memory 246-1may store a subset of the modules and data structures identified above.Furthermore, memory 246-1 may store additional modules and datastructures not described above. In some embodiments, the programs,modules, and data structures stored in memory 246-1, or thenon-transitory computer readable storage medium of memory 246-1, provideinstructions for implementing some of the methods described below. Insome embodiments, some or all of these modules may be implemented withspecialized hardware circuits that subsume part or all of the modulefunctionality.

Although FIG. 2B-1 shows system management module 151-1 in accordancewith some embodiments, FIG. 2B-1 is intended more as a functionaldescription of the various features which may be present in systemmanagement module 151-1 than as a structural schematic of theembodiments described herein. In practice, and as recognized by those ofordinary skill in the art, the programs, modules, and data structuresshown separately could be combined and some programs, modules, and datastructures could be separated.

FIG. 2B-2 is a block diagram illustrating a system management module151-2, in accordance with some embodiments, e.g., embodiments in whichthe system management module is located in the host, as shown in FIG.1B; in some such embodiments the storage system is called a host-managedstorage system. System management module 151-2 typically includes one ormore processing units (sometimes called CPUs or processors) 152-2 forexecuting modules, programs and/or instructions stored in memory 246-2and thereby performing processing operations, memory 246-2 (sometimescalled host memory), and one or more communication buses 248-2 forinterconnecting these components. The one or more communication buses248-2 optionally include circuitry (sometimes called a chipset) thatinterconnects and controls communications between system components.System management module 151-2 is coupled to storage system controller150 by the one or more communication buses 248-2. Memory 246-2 includeshigh-speed random access memory, such as DRAM, SRAM, DDR RAM or otherrandom access solid state memory devices, and may include non-volatilememory, such as one or more magnetic disk storage devices, optical diskstorage devices, flash memory devices, or other non-volatile solid statestorage devices. Memory 246-2 optionally includes one or more storagedevices remotely located from the CPU(s) 152-2. Memory 246-2, oralternatively the non-volatile memory device(s) within memory 246-2,comprises a non-transitory computer readable storage medium. In someembodiments, memory 246-2, or the non-transitory computer readablestorage medium of memory 246-2 stores the following programs, modules,and data structures, or a subset or superset thereof:

-   -   system mapping module 250-2 that is used for mapping virtual        logical addresses (e.g., used by computer system 142, FIG. 1B)        to intermediate addresses (e.g., which are mapped by storage        devices 160 to physical addresses, FIG. 1B);    -   metrics module 252-2 that is used for generating and/or        obtaining one or more metrics of a storage device (e.g., any of        storage devices 160-1 to 160-m, FIG. 1B);    -   trigger detection module 254-2 that is used for detecting a        trigger condition (e.g., in accordance with one or more metrics        of a storage device);    -   enabling module 256-2 that is used for enabling an amelioration        process associated with a trigger condition (e.g., detected by        trigger detection module 254-2);    -   notification module 258-2 that is used for notifying a host to        which a storage device is operatively coupled of a trigger        condition (e.g., detected by trigger detection module 254-2)        and/or of an absence of the trigger condition;    -   amelioration module 260-2 that is used for performing an        amelioration process to reduce declared capacity of non-volatile        memory of a storage device (e.g., storage device 160, FIG. 1B),        optionally including:        -   detection module 261-2 that is used for detecting an            amelioration trigger for reducing declared capacity of the            non-volatile memory of the storage device;        -   utilization module 262-2 that is used for reducing            utilization of the non-volatile memory of the storage            device; and        -   capacity module 264-2 that is used for reducing declared            capacity of the non-volatile memory of the storage device;    -   optionally, data redundancy module 266-2 that is used for        redundantly encoding data (e.g., to implement a particular RAID        (redundant array of independent disks) level); and    -   optionally, communication module 268-2 that is used for        facilitating communications with other devices, for example, via        a storage area network (SAN).

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and corresponds to a set ofinstructions for performing a function described above. The aboveidentified modules or programs (i.e., sets of instructions) need not beimplemented as separate software programs, procedures or modules, andthus various subsets of these modules may be combined or otherwisere-arranged in various embodiments. In some embodiments, memory 246-2may store a subset of the modules and data structures identified above.Furthermore, memory 246-2 may store additional modules and datastructures not described above. In some embodiments, the programs,modules, and data structures stored in memory 246-2, or thenon-transitory computer readable storage medium of memory 246-2, provideinstructions for implementing some of the methods described below. Insome embodiments, some or all of these modules may be implemented withspecialized hardware circuits that subsume part or all of the modulefunctionality.

Although FIG. 2B-2 shows system management module 151-2 in accordancewith some embodiments, FIG. 2B-2 is intended more as a functionaldescription of the various features which may be present in systemmanagement module 151-2 than as a structural schematic of theembodiments described herein. In practice, and as recognized by those ofordinary skill in the art, the programs, modules, and data structuresshown separately could be combined and some programs, modules, and datastructures could be separated.

FIG. 2C-1 is a block diagram illustrating a cluster management module181-1, in accordance with some embodiments, as shown in FIG. 1C. Clustermanagement module 181-1 typically includes one or more processing units(sometimes called CPUs or processors) 182-1 for executing modules,programs and/or instructions stored in memory 276-1 and therebyperforming processing operations, memory 276-1, and one or morecommunication buses 278-1 for interconnecting these components. The oneor more communication buses 278-1 optionally include circuitry(sometimes called a chipset) that interconnects and controlscommunications between system components. Cluster management module181-1 is coupled to additional module(s) 185 by the one or morecommunication buses 278-1. Memory 276-1 includes high-speed randomaccess memory, such as DRAM, SRAM, DDR RAM or other random access solidstate memory devices, and may include non-volatile memory, such as oneor more magnetic disk storage devices, optical disk storage devices,flash memory devices, or other non-volatile solid state storage devices.Memory 276-1 optionally includes one or more storage devices remotelylocated from the CPU(s) 182-1. Memory 276-1, or alternatively thenon-volatile memory device(s) within memory 276-1, comprises anon-transitory computer readable storage medium. In some embodiments,memory 276-1, or the non-transitory computer readable storage medium ofmemory 276-1 stores the following programs, modules, and datastructures, or a subset or superset thereof:

-   -   cluster mapping module 280-1 that is used for mapping virtual        logical addresses (e.g., used by computer system 172, FIG. 1C)        to intermediate addresses (e.g., which are mapped by storage        subsystems 192 to physical addresses, FIG. 1C);    -   metrics module 282-1 that is used for generating and/or        obtaining one or more metrics of a storage device (e.g., any of        the storage devices 194-1 to 194-n or 194-j to 194-k, FIG. 1C);    -   trigger detection module 284-1 that is used for detecting a        trigger condition (e.g., in accordance with one or more metrics        of a storage device);    -   enabling module 286-1 that is used for enabling an amelioration        process associated with a trigger condition (e.g., detected by        trigger detection module 284-1);    -   notification module 288-1 that is used for notifying a host to        which a storage device is operatively coupled of a trigger        condition (e.g., detected by trigger detection module 284-1)        and/or of an absence of the trigger condition;    -   amelioration module 290-1 that is used for performing an        amelioration process to reduce declared capacity of non-volatile        memory of a storage device (e.g., storage device 194, FIG. 1C),        optionally including:        -   detection module 291-1 that is used for detecting an            amelioration trigger for reducing declared capacity of the            non-volatile memory of the storage device;        -   utilization module 292-1 that is used for reducing            utilization of the non-volatile memory of the storage            device; and        -   capacity module 294-1 that is used for reducing declared            capacity of the non-volatile memory of the storage device;    -   optionally, data redundancy module 296-1 that is used for        redundantly encoding data; and    -   optionally, communication module 298-1 that is used for        facilitating communications with other devices, for example, via        a storage area network (SAN).

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and corresponds to a set ofinstructions for performing a function described above. The aboveidentified modules or programs (i.e., sets of instructions) need not beimplemented as separate software programs, procedures or modules, andthus various subsets of these modules may be combined or otherwisere-arranged in various embodiments. In some embodiments, memory 276-1may store a subset of the modules and data structures identified above.Furthermore, memory 276-1 may store additional modules and datastructures not described above. In some embodiments, the programs,modules, and data structures stored in memory 276-1, or thenon-transitory computer readable storage medium of memory 276-1, provideinstructions for implementing some of the methods described below. Insome embodiments, some or all of these modules may be implemented withspecialized hardware circuits that subsume part or all of the modulefunctionality.

Although FIG. 2C-1 shows cluster management module 181-1 in accordancewith some embodiments, FIG. 2C-1 is intended more as a functionaldescription of the various features which may be present in clustermanagement module 181-1 than as a structural schematic of theembodiments described herein. In practice, and as recognized by those ofordinary skill in the art, the programs, modules, and data structuresshown separately could be combined and some programs, modules, and datastructures could be separated.

FIG. 2C-2 is a block diagram illustrating a cluster management module181-2, in accordance with some embodiments, e.g., embodiments in whichthe cluster management module is located, at least in part, in the host,as shown in FIG. 1C; in some such embodiments the storage system useshost-based cluster management. Cluster management module 181-2 typicallyincludes one or more processing units (sometimes called CPUs orprocessors) 182-2 for executing modules, programs and/or instructionsstored in memory 276-2 and thereby performing processing operations,memory 276-2 (sometimes called host memory), and one or morecommunication buses 278-2 for interconnecting these components. The oneor more communication buses 278-2 optionally include circuitry(sometimes called a chipset) that interconnects and controlscommunications between system components. Cluster management module181-2 is coupled to cluster controller 180 by the one or morecommunication buses 278-2. Memory 276-2 includes high-speed randomaccess memory, such as DRAM, SRAM, DDR RAM or other random access solidstate memory devices, and may include non-volatile memory, such as oneor more magnetic disk storage devices, optical disk storage devices,flash memory devices, or other non-volatile solid state storage devices.Memory 276-2 optionally includes one or more storage devices remotelylocated from the CPU(s) 182-2. Memory 276-2, or alternatively thenon-volatile memory device(s) within memory 276-2, comprises anon-transitory computer readable storage medium. In some embodiments,memory 276-2, or the non-transitory computer readable storage medium ofmemory 276-2 stores the following programs, modules, and datastructures, or a subset or superset thereof:

-   -   cluster mapping module 280-2 that is used for mapping virtual        logical addresses (e.g., used by computer system 172, FIG. 1C)        to intermediate addresses (e.g., which are mapped by storage        subsystems 192 to physical addresses, FIG. 1C);    -   metrics module 282-2 that is used for generating and/or        obtaining one or more metrics of a storage device (e.g., any of        the storage devices 194-1 to 194-n or 194-j to 194-k, FIG. 1C);    -   trigger detection module 284-2 that is used for detecting a        trigger condition (e.g., in accordance with one or more metrics        of a storage device);    -   enabling module 286-2 that is used for enabling an amelioration        process associated with a trigger condition (e.g., detected by        trigger detection module 284-2);    -   notification module 288-2 that is used for notifying a host to        which a storage device is operatively coupled of a trigger        condition (e.g., detected by trigger detection module 284-2)        and/or of an absence of the trigger condition;    -   amelioration module 290-2 that is used for performing an        amelioration process to reduce declared capacity of non-volatile        memory of a storage device (e.g., storage device 194, FIG. 1C),        optionally including:        -   detection module 291-2 that is used for detecting an            amelioration trigger for reducing declared capacity of the            non-volatile memory of the storage device;        -   utilization module 292-2 that is used for reducing            utilization of the non-volatile memory of the storage            device; and        -   capacity module 294-2 that is used for reducing declared            capacity of the non-volatile memory of the storage device;    -   optionally, data redundancy module 296-2 that is used for        redundantly encoding data; and    -   optionally, communication module 298-2 that is used for        facilitating communications with other devices, for example, via        a storage area network (SAN).

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and corresponds to a set ofinstructions for performing a function described above. The aboveidentified modules or programs (i.e., sets of instructions) need not beimplemented as separate software programs, procedures or modules, andthus various subsets of these modules may be combined or otherwisere-arranged in various embodiments. In some embodiments, memory 276-2may store a subset of the modules and data structures identified above.Furthermore, memory 276-2 may store additional modules and datastructures not described above. In some embodiments, the programs,modules, and data structures stored in memory 276-2, or thenon-transitory computer readable storage medium of memory 276-2, provideinstructions for implementing some of the methods described below. Insome embodiments, some or all of these modules may be implemented withspecialized hardware circuits that subsume part or all of the modulefunctionality.

Although FIG. 2C-2 shows cluster management module 181-2 in accordancewith some embodiments, FIG. 2C-2 is intended more as a functionaldescription of the various features which may be present in clustermanagement module 181-2 than as a structural schematic of theembodiments described herein. In practice, and as recognized by those ofordinary skill in the art, the programs, modules, and data structuresshown separately could be combined and some programs, modules, and datastructures could be separated.

FIG. 2D is a block diagram illustrating amelioration module 230 includedin management module 121-1 of FIG. 2A-1 and/or management module 121-2of FIG. 2A-2, in accordance with some embodiments. As described abovewith respect to FIGS. 2A-1 and 2A-2, in some embodiments, ameliorationmodule 230 includes utilization module 232 and capacity module 234. Insome embodiments, utilization module 232 includes the following programsand/or modules, or a subset or superset thereof:

-   -   trimming module 235 that is used for trimming from a storage        device at least a portion of previously-written data that is no        longer used by a host (e.g., not live data 332, FIG. 3);    -   deleting module 236 that is used for deleting from a storage        device discardable data that is used by a host; and    -   moving module 237 that is used for moving a portion of data that        is used by a host from a storage device to another storage        device.

In some embodiments, capacity module 234 includes the following programsand/or modules, or a subset or superset thereof:

-   -   LBA reduction module 238 (sometimes called a logical address        reduction module) that is used for reducing a range of logical        addresses, reducing a count of logical addresses, and/or making        specific logical addresses unavailable to a host; and    -   advertising module 239 that is used for advertising a reduced        declared capacity of non-volatile memory of a storage device or        storage subsystem.

In some embodiments, the amelioration process includes a utilizationreduction process (e.g., performed by utilization module 232) and adeclared capacity reduction process (e.g., performed by capacity module234). In some embodiments, the amelioration process has a target reduceddeclared capacity to be achieved by the amelioration process, andutilizes the target reduced declared capacity to determine a targetamount of utilization reduction to be achieved by the ameliorationprocess. In some circumstances, such as when the amelioration processhas a target reduced declared capacity to be achieved by theamelioration process, and the amount of a storage device utilized isless than the target reduced declared capacity, the target amount ofutilization reduction is zero. In such circumstances, performance of theutilization reduction process, or one or more portions of theutilization reduction process, is unneeded and therefore skipped orforgone. Furthermore, in some embodiments, the amelioration process(e.g., periodically, semi-continuously, irregularly, initially and/orfinally) recomputes or re-evaluates a number of parameters, such as thetarget reduced declared capacity and/or the target amount of utilizationreduction, as those parameters may change in value due to theamelioration process and/or normal storage operations (e.g., read,write, erase and trim or unmap operations). In some circumstances, inaccordance with the recomputed or re-evaluated parameters, theutilization reduction is re-prioritized, re-scheduled, or aborted.Although FIG. 2D uses the example of amelioration module 230 included inFIGS. 2A-1 and 2A-2, the description of FIG. 2D similarly applies toother amelioration modules (e.g., amelioration module 260-1 of FIG.2B-1, amelioration module 260-2 of FIG. 2B-2, amelioration module 290-1of FIG. 2C-1, and/or amelioration module 290-2 of FIG. 2C-2), and forsake of brevity, the details are not repeated here.

The trim operation indicates that specific portions of the LBA space(320, FIG. 3) are converted to unallocated LBA space (340), therebyreducing utilization. The trim operation typically includes invalidatingone or more entries of a mapping table (e.g., mapping table 402, FIG. 4)used to translate logical addresses in a logical address space tophysical addresses in a physical address space of a storage device. As aresult of the trim operation, any data that was previously stored usingthe specific portion of the LBA space is no longer available to the host(e.g., is discarded). The physical pages used to previously store thisdata may be reused for other purposes. The reuse may be done coincidentin time with the trim operation or at a future time (e.g., duringgarbage collection). As discussed elsewhere, the reused physical pagesmay be used with a different encoding format or different redundancymechanism than the encoding format or different redundancy mechanismused by those same physical pages before the reuse. The trim operationis sometimes also referred to as an unmap operation. The trim operation,as used herein, is not necessarily identical to the trim operation ofthe SATA protocol. The unmap operation, as used herein, is notnecessarily identical to the unmap operation of the SCSI protocol.

FIG. 3 is a block diagram of a logical block address (LBA) space 320(sometimes called logical address (LA) space), in accordance with someembodiments. In some embodiments, a logical address is the address atwhich an item (e.g., a file or other data) resides from the perspectiveof a host (e.g., computer system 110, FIG. 1A, computer system 142, FIG.1B, and/or computer system 172, FIG. 1C). In some embodiments, a logicaladdress (e.g., in LBA space 320) differs from a physical address (e.g.,in physical space 318) due to the operation of a mapping function oraddress translator (e.g., a function or module that includes translationtable 212-1, FIG. 2A-1, or mapping table 402, FIG. 4). In someembodiments, a logical block address (LBA) is mapped to a physical flashaddress (e.g., a physical page number (PPN), including a bank, block,and page), as described further with respect to FIG. 4.

In some embodiments, a logical address space includes allocated logicaladdress space (e.g., allocated LBA space 342) and unallocated logicaladdress space (e.g., unallocated LBA space 340). In some embodiments,unallocated logical address space is logical address space at which nodata is stored. In some embodiments, unallocated logical address spaceincludes logical address space that has never been written to and/or hasbeen discarded (previously written data may be discarded through a trimor unmap operation, and is sometimes called trimmed logical addressspace). For example, in FIG. 3, unallocated LBA space 340 includestrimmed LBA space 330. In some embodiments, allocated logical addressspace is logical address space that was previously-written by a host,the previously-written data including data that is no longer used by ahost (e.g., not live data 332) and data that is still in use by the host(e.g., live data 334). In some embodiments, not live data is data in aportion of the logical address space that is marked as free, availableor unused in the metadata of a file system. Optionally, a file systemmay choose to convert not live address space into unallocated addressspace through a trim or unmap operation.

In FIG. 3, allocated LBA space 342 represents an amount of allocatedspace, and unallocated LBA space 340 represents an amount of unallocatedspace. However, neither allocated LBA space 342 nor unallocated LBAspace 340 is necessarily a single contiguous region of LBA space 320.Similarly, live data 334 and not live data 332 in FIG. 3 representamounts (e.g., counts of LBAs) of live data and not live data,respectively. However, neither live data 334 nor not live data 332 isnecessarily a single contiguous region of LBA space 320 or allocated LBAspace 342, nor do the positions of live data 334 and not live data 332illustrated in FIG. 3 have any correlation to the logical or physicaladdress values of the live data and not live data. Typically, live data334 and/or not live data 332 will be present in multiple regions of LBAspace 320, and are thus non-contiguous. Optionally, however, a remappingor coalescing process, which can also be called defragmentation, can beperformed to consolidate some or all live data 334 into a contiguousregion of LBA space 320.

Allocated logical address space (342) is space that is utilized. Theutilization reduction modules and processes discussed herein aremodules, applications and processes whose purpose is to reduce the sizeof the allocated logical address space, and thus reduce utilization ofnon-volatile memory in a storage device or data storage system.Typically, reducing the size of the allocated logical address spacerequires reducing the amount of live data 334 and/or not live data 332stored by a storage device, or storage system, thereby converting aportion of the allocated logical address space into unallocated logicaladdress space. In some embodiments, portions of not live data 332 aretrimmed, and thereby converted into unallocated logical address spacethrough the use of trim or unmap operations.

In some embodiments, a logical address may be outside of LBA Space (320)and is therefore unavailable. A previously available logical address canbe made unavailable by reducing the size of the LBA space (320) suchthat that address is no longer within LBA space (320) and hence becomesunavailable (e.g. it is an undefined operation or erroneous operation torequest a normal storage operation to a logical address that is outsideof LBA space (320)). As noted above, LBA Space (320) can be reduced by acommand to the storage device, or a host can limit its usage of logicaladdresses to a reduced range of logical addresses therefore effectivelyreducing LBA space (320).

In some embodiments, the total number of allocated logical addresses(342) is limited. In such embodiments, specific logical addresses areconsidered to be unavailable if usage of them would cause the system toexceed the limited total number. For example, if the total number ofallocated logical addresses is limited to five and the currentlyallocated addresses are 1, 3, 19, 45 and 273838 then any specificlogical address other than these five (e.g., 6, 15, 137, etc.) would beconsidered unavailable.

FIG. 4 is a block diagram of a mapping table 402 and physical addressspace 410, in accordance with some embodiments. In some embodiments,mapping table 402 is used to translate a logical block address (LBA)from the perspective of a host (e.g., computer system 110, FIG. 1A) to aphysical address in a physical address space (e.g., physical addressspace 410) of non-volatile memory in a storage device (e.g., storagedevice 120, FIG. 1A). In some embodiments, an LBA is the address of thesmallest unit of stored data that is addressable by a host (e.g., 512 Bor 4096 B). In some embodiments, LBAs are a sequence of integersorganized in numerical order in the logical address space. In someembodiments, LBAs are integers chosen from a logical address space butneed not be contiguous. For example, in implementations that utilize asparse logical address space, the amount of addressable space isgoverned by a limit on the number of logical addresses that can beallocated, but those logical addresses are distributed over a largeraddress space than the maximum number of logical addresses that can beallocated (e.g., to a host or a set of hosts or clients).

In some embodiments, mapping table 402 is stored in memory associatedwith the storage device (e.g., in memory 206-1, as part of translationtable 212-1, FIG. 2A-1). In some embodiments, a physical address is aphysical page number (PPN), including a bank number, a block number, anda page number. In the example shown in FIG. 4, LBA 0 is mapped to bank 1(e.g., Bank 420-1), block 3 (e.g., Block 421-3), page 37 (pages notshown in FIG. 4) of physical address space 410. FIG. 4 shows thatphysical address space 410 includes a plurality of non-volatile memoryblocks 421, 422 423, 424. As described above, and as shown in therepresentation of block 424-p, each non-volatile memory block in thephysical address space of a storage device typically includes aplurality of pages 426, where each page is typically an instance of thesmallest individually accessible (e.g., readable or writable) portion ofa block. Although FIG. 4 illustrates one example of a logical address tophysical address mapping, in other embodiments, different mappings maybe used. For example, in some embodiments, each of the logical addressentries corresponds to multiple (e.g., eight) logical addresses (e.g., 8LBAs per logical address entry). In some embodiments mapping table 402need not contain contiguous LBA addresses and may be organized in anymanner to facilitate lookup operations, e.g., hash table, binary tree,content addressable memory, and others.

As discussed below with reference to FIG. 5A, a single-level flashmemory cell (SLC) stores one bit (“0” or “1”). Thus, the storage densityof a SLC memory device is one bit of information per memory cell. Amulti-level flash memory cell (MLC), however, can store two or more bitsof information per cell by using different ranges within the totalvoltage range of the memory cell to represent a multi-bit bit-tuple. Inturn, the storage density of a MLC memory device is multiple-bits percell (e.g., two bits per memory cell).

Flash memory devices utilize memory cells to store data as electricalvalues, such as electrical charges or voltages. Each flash memory celltypically includes a single transistor with a floating gate that is usedto store a charge, which modifies the threshold voltage of thetransistor (i.e., the voltage needed to turn the transistor on). Themagnitude of the charge, and the corresponding threshold voltage, isused to represent one or more data values. In some embodiments, during aread operation, a reading threshold voltage is applied to the controlgate of the transistor and the resulting sensed current or voltage ismapped to a data value.

The terms “cell voltage” and “memory cell voltage,” in the context offlash memory cells, typically means the threshold voltage of the memorycell, which is the minimum voltage that needs to be applied to the gateof the memory cell's transistor in order for the transistor to conductcurrent. Similarly, reading threshold voltages (sometimes also calledreading signals, reading voltages, and/or read thresholds) applied to aflash memory cells are gate voltages applied to the gates of the flashmemory cells to determine whether the memory cells conduct current atthat gate voltage. In some embodiments, when a flash memory cell'stransistor conducts current at a given reading threshold voltage,indicating that the cell voltage is less than the reading thresholdvoltage, the raw data value for that read operation is a “1,” andotherwise the raw data value is a “0.”

FIG. 5A is a simplified, prophetic diagram of voltage distributions 300a found in a single-level flash memory cell (SLC) over time, inaccordance with some embodiments. The voltage distributions 300 a shownin FIG. 5A have been simplified for illustrative purposes. In thisexample, the SLC's voltage range extends approximately from a voltage,V_(SS), at a source terminal of an NMOS transistor to a voltage, V_(DD),at a drain terminal of the NMOS transistor. As such, voltagedistributions 300 a extend between V_(SS) and V_(DD).

Sequential voltage ranges 301 and 302 between source voltage V_(SS) anddrain voltage V_(DD) are used to represent corresponding bit values “1”and “0,” respectively. Each voltage range 301, 302 has a respectivecenter voltage V₁ 301 b, V₀ 302 b. As described below, in manycircumstances the memory cell current sensed in response to an appliedreading threshold voltages is indicative of a memory cell voltagedifferent from the respective center voltage V₁ 301 b or V₀ 302 bcorresponding to the respective bit value written into the memory cell.Errors in cell voltage, and/or the cell voltage sensed when reading thememory cell, can occur during write operations, read operations, or dueto “drift” of the cell voltage between the time data is written to thememory cell and the time a read operation is performed to read the datastored in the memory cell. For ease of discussion, these effects arecollectively described as “cell voltage drift.” Each voltage range 301,302 also has a respective voltage distribution 301 a, 302 a that mayoccur as a result of any number of a combination of error-inducingfactors, examples of which are identified above.

In some implementations, a reading threshold voltage V_(R) is appliedbetween adjacent center voltages (e.g., applied proximate to the halfwayregion between adjacent center voltages V₁ 301 b and V₀ 302 b).Optionally, in some implementations, the reading threshold voltage islocated between voltage ranges 301 and 302. In some implementations,reading threshold voltage V_(R) is applied in the region proximate towhere the voltage distributions 301 a and 302 a overlap, which is notnecessarily proximate to the halfway region between adjacent centervoltages V₁ 301 b and V₀ 302 b.

In order to increase storage density in flash memory, flash memory hasdeveloped from single-level (SLC) cell flash memory to multi-level cell(MLC) flash memory so that two or more bits can be stored by each memorycell. As discussed below with reference to FIG. 5B, a MLC flash memorydevice is used to store multiple bits by using voltage ranges within thetotal voltage range of the memory cell to represent differentbit-tuples. A MLC flash memory device is typically more error-prone thana SLC flash memory device created using the same manufacturing processbecause the effective voltage difference between the voltages used tostore different data values is smaller for a MLC flash memory device.Moreover, due to any number of a combination of factors, such aselectrical fluctuations, defects in the storage medium, operatingconditions, device history, and/or write-read circuitry, a typical errorincludes a stored voltage level in a particular MLC being in a voltagerange that is adjacent to the voltage range that would otherwise berepresentative of the correct storage of a particular bit-tuple. Asdiscussed in greater detail below with reference to FIG. 5B, the impactof such errors can be reduced by gray-coding the data, such thatadjacent voltage ranges represent single-bit changes between bit-tuples.

FIG. 5B is a simplified, prophetic diagram of voltage distributions 300b found in a multi-level flash memory cell (MLC) over time, inaccordance with some embodiments. The voltage distributions 300 b shownin FIG. 5B have been simplified for illustrative purposes. The cellvoltage of a MLC approximately extends from a voltage, V_(SS), at thesource terminal of a NMOS transistor to a voltage, V_(DD), at the drainterminal. As such, voltage distributions 300 b extend between V_(SS) andV_(DD).

Sequential voltage ranges 311, 312, 313, 314 between the source voltageV_(SS) and drain voltages V_(DD) are used to represent correspondingbit-tuples “11,” “01,” “00,” “10,” respectively. Each voltage range 311,312, 313, 314 has a respective center voltage 311 b, 312 b, 313 b, 314b. Each voltage range 311, 312, 313, 314 also has a respective voltagedistribution 311 a, 312 a, 313 a, 314 a that may occur as a result ofany number of a combination of factors, such as electrical fluctuations,defects in the storage medium, operating conditions, device history(e.g., number of program-erase (P/E) cycles), and/or imperfectperformance or design of write-read circuitry.

Ideally, during a write operation, the charge on the floating gate ofthe MLC would be set such that the resultant cell voltage is at thecenter of one of the ranges 311, 312, 313, 314 in order to write thecorresponding bit-tuple to the MLC. Specifically, the resultant cellvoltage would be set to one of V₁₁ 311 b, V₀₁ 312 b, V₀₀ 313 b and V₁₀314 b in order to write a corresponding one of the bit-tuples “11,”“01,” “00” and “10.” In reality, due to the factors mentioned above, theinitial cell voltage may differ from the center voltage for the datawritten to the MLC.

Reading threshold voltages V_(RA), V_(RB) and V_(RC) are positionedbetween adjacent center voltages (e.g., positioned at or near thehalfway point between adjacent center voltages) and, thus, definethreshold voltages between the voltage ranges 311, 312, 313, 314. Duringa read operation, one of the reading threshold voltages V_(RA), V_(RB)and V_(RC) is applied to determine the cell voltage using a comparisonprocess. However, due to the various factors discussed above, the actualcell voltage, and/or the cell voltage received when reading the MLC, maybe different from the respective center voltage V₁₁ 311 b, V₀₁ 312 b,V₀₀ 313 b or V₁₀ 314 b corresponding to the data value written into thecell. For example, the actual cell voltage may be in an altogetherdifferent voltage range, strongly indicating that the MLC is storing adifferent bit-tuple than was written to the MLC. More commonly, theactual cell voltage may be close to one of the read comparison voltages,making it difficult to determine with certainty which of two adjacentbit-tuples is stored by the MLC.

Errors in cell voltage, and/or the cell voltage received when readingthe MLC, can occur during write operations, read operations, or due to“drift” of the cell voltage between the time data is written to the MLCand the time a read operation is performed to read the data stored inthe MLC. For ease of discussion, sometimes errors in cell voltage,and/or the cell voltage received when reading the MLC, are collectivelycalled “cell voltage drift.”

One way to reduce the impact of a cell voltage drifting from one voltagerange to an adjacent voltage range is to gray-code the bit-tuples.Gray-coding the bit-tuples includes constraining the assignment ofbit-tuples such that a respective bit-tuple of a particular voltagerange is different from a respective bit-tuple of an adjacent voltagerange by only one bit. For example, as shown in FIG. 5B, thecorresponding bit-tuples for adjacent ranges 301 and 302 arerespectively “11” and “01,” the corresponding bit-tuples for adjacentranges 302 and 303 are respectively “01” and “00,” and the correspondingbit-tuples for adjacent ranges 303 and 304 are respectively “00” and“10.” Using gray-coding, if the cell voltage drifts close to a readcomparison voltage level, the error is typically limited to a single bitwithin the 2-bit bit-tuple.

Although the description of FIG. 5B uses an example in which q=2 (i.e.,2 bits per cell in a MLC flash memory), those skilled in the art willappreciate that the embodiments described herein may be extended tomemory cells that have more than four possible states per cell, yieldingmore than two bits of information per cell. For example, in someembodiments, a triple-level memory cell (TLC) has eight possible statesper cell, yielding three bits of information per cell. As anotherexample, in some embodiments, a quad-level memory cell (QLC) has 16possible states per cell, yielding four bits of information per cell. Asanother example, in some embodiments, a cell might store only 6 states,yielding approximately 2.5 bits of information per cell, meaning thattwo cells together would provide 36 possible states, more thansufficient to store 5 bits of information per pair of cells.

FIG. 6 illustrates a flowchart representation of a method 600 ofmanaging a storage system, in accordance with some embodiments. At leastin some embodiments, method 600 is performed by a storage device (e.g.,storage device 120, FIG. 1A) or one or more components of the storagedevice (e.g., storage controller 124 and/or storage medium 130, FIG.1A), wherein the storage device is operatively coupled with a hostsystem (e.g., computer system 110, FIG. 1A). In some embodiments, method600 is governed by instructions that are stored in a non-transitorycomputer readable storage medium and that are executed by one or moreprocessors of a device, such as the one or more processing units (CPUs)122-1 of management module 121-1, shown in FIGS. 1A and 2A-1. In someembodiments, method 600 is performed by a storage system (e.g., datastorage system 100, FIG. 1A, data storage system 140, FIG. 1B, and/ordata storage system 170, FIG. 1C) or one or more components of thestorage system (e.g., computer system 110 and/or storage device 120,FIG. 1A, storage system controller 150, FIG. 1B, and/or clustercontroller 180, FIG. 1C). In some embodiments, some of the operations ofmethod 600 are performed at a host (e.g., computer system 110, FIG. 1A,computer system 142, FIG. 1B, and/or computer system 172, FIG. 1C) andinformation is transmitted to a storage device (e.g., storage device120, FIG. 1A) and/or one or more subsystems of a storage system (e.g.,storage system controller 150, FIG. 1B, and/or cluster controller 180,FIG. 1C). In some embodiments, method 600 is governed, at least in part,by instructions that are stored in a non-transitory computer readablestorage medium and that are executed by one or more processors of asubsystem of a storage system, such as the one or more processing units(CPUs) 152-1 of system management module 151-1, shown in FIGS. 1B and2B-1 or the one or more processing units (CPUs) 182-1 of clustermanagement module 181-1, shown in FIGS. 1C and 2C-1. In someembodiments, method 600 is governed, at least in part, by instructionsthat are stored in a non-transitory computer readable storage medium andthat are executed by one or more processors of a host (processors ofhost not shown in FIGS. 1A, 1B, and 1C). For ease of explanation, thefollowing describes method 600 as performed by a storage device (e.g.,storage device 120, FIG. 1A). However, those skilled in the art willappreciate that in other embodiments, one or more of the operationsdescribed in method 600 are performed by a host (e.g., computer system110, FIG. 1A, computer system 142, FIG. 1B, and/or computer system 172,FIG. 1C) and/or one or more subsystems of a storage system distinct fromthe storage device (e.g., storage system controller 150, FIG. 1B, and/orcluster controller 180, FIG. 1C).

A storage device (e.g., storage device 120, FIG. 1A), triggers (602) acondition for reducing declared capacity of non-volatile memory of thestorage device in accordance with one or more metrics of the storagedevice (e.g., including one or more status metrics corresponding to thestorage device's ability to retain data, one or more performance metricscorresponding to performance of the storage device, one or more wearmetrics corresponding to wear on the storage device, and/or one or moretime metrics). Metrics of the storage device include metrics (e.g., wearmetrics such as P/E cycle counts, write operation counts and the like)of the non-volatile storage media (e.g., storage medium 130, FIG. 1A) ofthe storage device, but are not necessarily limited to such metrics. Forexample, some metrics (e.g., some performance metrics, such as latencymetrics, metrics that measure how long it takes or how many operationsare required to complete a write or erase operation, etc.) of thestorage device reflect both storage media performance as well ascontroller and/or other storage device component performance.

In some embodiments, the metrics of the storage device used to determinethe trigger condition include a write amplification metric of thestorage device. Another metric of the storage device that is used, insome embodiments, to determine the trigger condition is anover-provisioning metric (e.g., quantity or percentage of total storagecapacity that is in excess of the declared capacity of the storagedevice, and/or quantity or percentage of total storage capacity that isin excess of the declared capacity of the storage device after aprojected conversion of a number of memory blocks (or other portions ofthe storage device) from a current encoding format (e.g., TLC, MLCand/or data redundancy mechanism) to a lower storage density encodingformat (e.g., MLC, SLC and/or data redundancy mechanism). For example,in some embodiments, a trigger condition is determined if a projectedover-provisioning metric, corresponding to a number of blocks (or otherportions) of the storage device removed from service (e.g., that havebeen or will be removed from service) due to wear or due to failure ofthose blocks (or other portions) to meet a predefined quality of servicemetric, falls below a predefined threshold (e.g., a non-zero thresholdsuch as 2 percent or 5 percent or the like), or falls below a thresholddetermined in accordance with a measured or projected writeamplification of the storage device.

Write amplification is a phenomenon where the actual amount of physicaldata written to a storage medium (e.g., storage medium 130 in storagedevice 120) is a multiple of the logical amount of data written by ahost (e.g., computer system 110, sometimes called a host) to the storagemedium. As discussed above, when a block of storage medium must beerased before it can be re-written, the garbage collection process toperform these operations results in re-writing data one or more times.This multiplying effect increases the number of writes required over thelife of a storage medium, which shortens the time it can reliablyoperate. The formula to calculate the write amplification of a storagesystem is given by equation:

$\frac{{{amount}\mspace{14mu} {of}\mspace{14mu} {data}\mspace{14mu} {written}\mspace{14mu} {to}\mspace{14mu} a\mspace{14mu} {storage}\mspace{14mu} {medium}}\;}{{amount}\mspace{14mu} {of}\mspace{14mu} {data}\mspace{14mu} {written}\mspace{14mu} {by}\mspace{14mu} a\mspace{14mu} {host}}$

One of the goals of any flash memory based data storage systemarchitecture is to reduce write amplification as much as possible sothat available endurance is used to meet storage medium reliability andwarranty specifications. Higher system endurance also results in lowercost as the storage system may need less over-provisioning. By reducingwrite amplification, the endurance of the storage medium is increasedand the overall cost of the storage system is decreased. Generally,garbage collection is performed on erase blocks with the fewest numberof valid pages for best performance and best write amplification.

In some embodiments, the trigger condition is detected in accordancewith a non-linear and/or linear combination of the one or more metrics.For example, in some embodiments, the trigger condition is detected bycomparing a wear metric such as P/E cycle counts to a previouslydetermined value, e.g., a threshold value. In some embodiments, thetrigger condition can also be asserted by other means, e.g., by a humanoperator or scheduled by a human operator. For example, it may bedesirable to initiate the amelioration process because of the expectedavailability or unavailability of resources. For example, it may bedesirable to initiate the amelioration process because performancecharacteristics of the storage device (including reliability) arealtered.

In some embodiments, the trigger condition is detected in accordancewith historical knowledge of the one or more metrics. For example,historical knowledge can be a running average of one or more metrics. Inanother example, historical knowledge can be used to determine (e.g.,compute) one or more projected values of one or more metrics at aparticular time in the future (e.g., an hour, day, week, or month in thefuture), and the trigger condition can be detected in accordance withthe one or more projected values. The latter methodology can beparticularly useful for avoiding events that result in loss of data(e.g., due to wear out), or more generally for avoiding events thatsignificantly impact on the quality of service provided by a storagesystem, and for enabling a storage system to undertake ameliorativemeasures prior to there being an urgent need to do so. For example, insome embodiments, the trigger condition is detected by comparing ahistorical wear metric such as P/E cycle counts to a previouslydetermined value to anticipate wear out of a portion of the storagemedia. Similarly, in some embodiments, the trigger condition is detectedby comparing a historical metric, such as the bit error rate (BER), orthe rate of change of the metric, BER (of the storage media, or aportion of the storage media), or a projected value (e.g., a projectedBER rate at a particular time in the future, as determined based on acurrent or historical BER and a rate of change of the BER), against apreviously determined value to anticipate performance degradation due toincreased computation requirements of error correction.

In a storage system with a plurality of storage devices the triggercondition may be dependent on metrics obtained from a plurality of thestorage devices. The amelioration process may operate on more than onestorage device at a time, either sequentially or in parallel. Forexample, a storage system may have a fixed maximum rate of capacityreduction independent of how many storage devices are currently beingoperated on in parallel by the amelioration process (e.g., maximum rateof data movement between the storage devices while reducingutilization). The trigger condition should include considering,separately and in combination, the metrics of the plurality of storagedevices when determining the targeted capacity reduction and, due to thefixed maximum rate, the scheduling of the amelioration process.

The storage device notifies (604) a host (e.g., computer system 110,FIG. 1A, computer system 142, FIG. 1B, computer system 172, FIG. 1C,storage system controller 150, FIG. 1B, and/or cluster controller 180,FIG. 1C) to which the storage device is operatively coupled of thetrigger condition for reducing declared capacity of the non-volatilememory of the storage device. In some embodiments, upon receipt of thenotification of the trigger condition the host sends an ameliorationtrigger to initiate the amelioration process (606).

The storage device or a host detects the amelioration trigger and, inaccordance with the detected amelioration trigger, performs anamelioration process (606) to reduce declared capacity of thenon-volatile memory of the storage device. In some embodiments, theamelioration process includes a process to reduce utilization (608), aprocess to reduce declared capacity (610), and/or a process to advertise(612) a reduced declared capacity. As described above with respect toFIG. 2D, in some embodiments, the amelioration process (606) includes autilization reduction process (608) (e.g., performed by utilizationmodule 232, FIGS. 2A-1 and 2A-2, utilization module 262, FIGS. 2B-1 and2B-2, or utilization module 292, FIGS. 2C-1 and 2C-2) and/or a declaredcapacity reduction process (610) (e.g., performed by capacity module234, FIGS. 2A-1 and 2A-2, capacity module 264, FIGS. 2B-1 and 2B-2, orcapacity module 294, FIGS. 2C-1 and 2C-2). In some circumstances, suchas when the amount of the storage device utilized by the host is lessthan the reduced declared capacity, performance of the process to reduceutilization (608), or one or more portions of the process to reduceutilization, is skipped or forgone. Although FIG. 6 shows operations608, 610, and 612 as sequential processes of the amelioration process(606), in some embodiments, these processes may be overlapped,non-sequential, and/or performed in a different order.

In some embodiments, prior to the operations described above in FIG. 6(e.g., operations 602, 604, and/or 606), method 600 includes reducing(601) over-provisioning of the non-volatile memory of the storagedevice. As described above, over-provisioning refers to a quantity orpercentage of total storage capacity that is in excess of the declaredcapacity of the storage device. In some embodiments, over-provisioningrefers to the difference between the physical capacity of the storagedevice (e.g., the physical capacity less capacity set aside formanagement data structures and metadata) for storing user data (e.g.,data stored in the storage system on behalf of a host or host system),and the logical capacity presented as available for a host or user. Forexample, in some embodiments, if a non-volatile memory of a storagedevice has 12 GB of total storage capacity (e.g., total storage capacityfor storing user data) and 10 GB of declared capacity, then thenon-volatile memory of the storage device has 2 GB of over-provisioning.Unlike declared capacity, which is the storage capacity available to ahost, the extra capacity of over-provisioning is not visible to the hostas available storage. Instead, over-provisioning is used to increaseendurance of a storage device (e.g., by distributing the total number ofwrites and erases across a larger population of blocks and/or pages overtime), improve performance (e.g., by providing additional buffer spacefor managing P/E cycles and improving the probability that a writeoperation will have immediate access to a pre-erased block), and reducewrite amplification.

In some embodiments, reducing (601) over-provisioning includes: (1)detecting a first wear condition of non-volatile memory of a storagedevice of a storage system, wherein a total storage capacity of thenon-volatile memory of the storage device includes declared capacity andover-provisioning, and (2) in response to detecting the first wearcondition, performing a remedial action that reduces over-provisioningof the non-volatile memory of the storage device without reducingdeclared capacity of the non-volatile memory of the storage device. Insome embodiments, performing a remedial action that reducesover-provisioning includes marking one or more blocks of thenon-volatile memory as unusable. In some embodiments, performing aremedial action that reduces over-provisioning includes converting oneor more MLC blocks to SLC, or more generally, changing the physicalencoding format of one or more blocks of the non-volatile memory. Insome embodiments, reducing over-provisioning is performed by anover-provisioning module of management module 121, system managementmodule 151, or cluster management module 181 (e.g., in memory 206 ofFIGS. 2A-1 and 2A-2, in memory 246 of FIGS. 2B-1 and 2B-2, or in memory276 of FIGS. 2C-1 and 2C-2, respectively, but not explicitly shown).Furthermore, in some circumstances or in some embodiments,over-provisioning reducing operation 601 is performed multiple timesprior to the first time operation 602 is performed. For example,over-provisioning reducing operation 601 may be repeated each ofmultiple times that a predefined wear condition is detected, untilover-provisioning falls to or below a predefined minimum level.

In some embodiments, the first wear condition is detected in accordancewith one or more metrics of the storage device (e.g., including one ormore status metrics corresponding to the storage device's ability toretain data, one or more performance metrics corresponding toperformance of the storage device, one or more wear metricscorresponding to wear on the storage device, and/or one or more timemetrics), as described above with respect to operation 602. In someembodiments, the first wear condition is detected in accordance with adetermination that the one or more metrics of the storage device satisfya first criterion and over-provisioning of the non-volatile memory ofthe storage device is greater than a predefined threshold (e.g., 2percent of the declared capacity, at least 100 blocks, or 40 blocks+n %of declared capacity, etc.).

In some embodiments, detecting the trigger condition (as described abovewith respect to operation 602) comprises detecting a second wearcondition distinct from the first wear condition. For example, in someembodiments, the trigger condition (or the second wear condition) isdetected in accordance with a determination that the one or more metricsof the storage device satisfy a second criterion (e.g., the firstcriterion used for the first wear condition or another criterion) andover-provisioning of the non-volatile memory of the storage device isless than or equal to (e.g., not greater than) a predefined threshold(e.g., 2 percent of the declared capacity, at least 100 blocks, or 40blocks+n % of declared capacity, etc.).

FIGS. 7A-7D illustrate a flowchart representation of a method 700 ofmanaging a storage system, in accordance with some embodiments. In someembodiments, the method 700 includes detecting (702) an ameliorationtrigger for reducing declared capacity of non-volatile memory of astorage device (e.g., storage device 120 of FIG. 1A, or any of storagedevices 160-1 to 160-m of FIG. 1B, or any of storage devices 194-1 to194-n or 194-j to 194-k of FIG. 1C) of the storage system (e.g., datastorage system 100, FIG. 1A; data storage system 140, FIG. 1B; or datastorage system 170, FIG. 1C), and in accordance with the detectedamelioration trigger, performing (704) an amelioration process to reducedeclared capacity of the non-volatile memory of the storage device, theperforming including: altering an encoding format of at least a portionof the non-volatile memory of the storage device, and reducing declaredcapacity of the non-volatile memory of the storage device. The encodingformat of data comprises the physical encoding of data in the storagemedia combined with the logical encoding of data which uses one or moreof the redundancy mechanisms as described above. The physical encodingformat of a portion of non-volatile memory is a physical encoding formatof the memory cells comprising the portion of non-volatile memory, andindicates the number of bits of information that can be stored permemory cell. As described above with respect to FIGS. 5A and 5B,examples of physical encoding formats include a single-level flashmemory cell (SLC), a multi-level flash memory cell (MLC), a triple-levelmemory cell (TLC), and a quad-level memory cell (QLC), capable ofstoring up to one, two, three, and four bits per memory cell,respectively. The physical encoding format can further include otherphysical encoding formats for a non-integer number of bits ofinformation per memory cell. Altering of the encoding format may involvealtering the physical encoding format or the redundancy mechanism(s) orboth the physical encoding format and the redundancy mechanism(s). Insome embodiments, altering of the encoding format is a multi-stepsequence. In a first step of this sequence an indication of the newencoding format is made or recorded. This indication will informsubsequent operations to use the indicated (e.g., lower density)encoding format. In a subsequent step, e.g., garbage collection, datawill be recorded with the new encoding format. In some embodiments,reducing declared capacity of non-volatile memory of the storage deviceincludes reducing storage capacity, of the non-volatile memory of thestorage device, available to the host. The declared capacity ofnon-volatile memory of the storage device is sometimes called theadvertised capacity, and is typically used by the operating systemand/or a file system of the host as the maximum capacity that the host'soperating system or file system will attempt to allocate.

In some embodiments, the amelioration trigger is detected (702) by anamelioration module (e.g., amelioration module 230 of FIG. 2A-1, 2A-2 or2D, 260 of FIG. 2B-1 or 2B-2, or 290 of FIG. 2C-1 or 2C-2), or acomponent thereof (e.g., detection module 231 of FIG. 2A-1, 2A-2 or 2D,261 of FIG. 2B-1 or 2B-2, or 291 of FIG. 2C-1 or 2C-2). In someembodiments, detecting the amelioration trigger includes receiving orgenerating the amelioration trigger. Furthermore, in some embodiments,prior to detecting the amelioration trigger (702), method 700 includesdetecting a wear condition and reducing over-provisioning of thenon-volatile memory of the storage device, without reducing declaredcapacity of the non-volatile memory of the storage device, as describedabove with respect to operation 601 of FIG. 6.

In some embodiments, an amelioration module (e.g., amelioration module230, FIGS. 2A-1 and 2A-2, amelioration module 260, FIGS. 2B-1 and 2B-2,and/or amelioration module 290, FIGS. 2C-1 and 2C-2) is used to perform(704) the amelioration process, in accordance with the ameliorationtrigger, to reduce declared capacity of the non-volatile memory of thestorage device. In some embodiments, the amelioration module includes amodule (e.g., utilization module 232, FIG. 2D) for altering an encodingformat of at least a portion of the non-volatile memory of the storagedevice, and a capacity module (e.g., capacity module 234) for reducingdeclared capacity of the non-volatile memory of the storage device.

In some embodiments, the detecting, the performing, or both thedetecting and the performing are performed (706) by the storage device(e.g., storage device 120, FIG. 1A, storage device 160, FIG. 1B, orstorage device 194, FIG. 1C) or one or more components of the storagedevice (e.g., storage controller 124, FIG. 1A). In some embodiments,method 700, or at least the detecting operation 702 and/or performingoperation 704 of method 700, is governed by instructions that are storedin a non-transitory computer readable storage medium and that areexecuted by one or more processors of a device, such as the one or moreprocessing units (CPUs) 122-1 of management module 121-1, shown in FIGS.1A and 2A-1.

In some embodiments, the detecting, the performing, or both thedetecting and the performing are performed (708) by one or moresubsystems of the storage system distinct from the storage device. Forexample, in some of these embodiments, the detecting, the performing, orboth the detecting and the performing are performed by a storage systemcontroller (e.g., storage system controller 150, FIG. 1B) of the storagesystem (e.g., data storage system 140, FIG. 1B). In some embodiments,method 700, or at least the detecting operation 702 and/or performingoperation 704 of method 700, is governed by instructions that are storedin a non-transitory computer readable storage medium and that areexecuted by one or more processors of a device, such as the one or moreprocessing units (CPUs) 152-1 of management module 151-1, shown in FIGS.1B and 2B-1.

In some embodiments, the detecting, the performing, or both thedetecting and the performing are performed (710), at least in part, bythe host. In some embodiments, method 700, or at least the detectingoperation 702 and/or performing operation 704 of method 700, is governedat least in part by instructions that are stored in a non-transitorycomputer readable storage medium and that are executed by one or moreprocessors of a host (processors not shown in FIGS. 1A, 1B and 1C), suchas the one or more processing units (CPUs) of management module 121-2(FIGS. 1A and 2A-2); or management module 151-2 (FIGS. 1B and 2B-2); ormanagement module 181-2 (FIGS. 1C and 2C-2).

In some embodiments, the host (e.g., computer system 110 of FIG. 1A,computer system 142 of FIG. 1B, or computer system 172 of FIG. 1C)includes (712) a client on behalf of which data is stored in the storagesystem (e.g., data storage system 100, FIG. 1A; data storage system 140,FIG. 1B; data storage system 170, FIG. 1C). In some embodiments, theclient is or includes an entity on behalf of which data is stored in thestorage system.

In some embodiments, the host includes (714) a storage system controller(e.g., storage system controller 150, FIG. 1B) of the storage system(e.g., data storage system 140, FIG. 1B). In some of these embodiments,the storage system controller controls and/or coordinates operationsamong one or more storage devices. In some of these embodiments, thedata storage system (e.g., data storage system 140, FIG. 1B) is called ascale-up system.

In some embodiments, the host includes (716) a cluster controller (e.g.,cluster controller 180, FIG. 1C) of the storage system (e.g., datastorage system 170, FIG. 1C). In some of these embodiments, the clustercontroller controls and/or coordinates operations among one or more datastorage subsystems, as shown for example in FIG. 1C, where each of thedata storage subsystems may be implemented as a data storage systemhaving one or more storage devices (e.g., data storage system 140, FIG.1B). In some of these embodiments, the data storage system (e.g., datastorage system 170, FIG. 1C) is called a scale-out system or a clusteredstorage system.

In some embodiments, altering (704) the encoding format of at least aportion of the non-volatile memory of the storage device includesaltering (718) the encoding format of at least a portion of thenon-volatile memory of the storage device in accordance with one or moreparameters for the amelioration process. In some embodiments, storagecontroller 124 (FIG. 1A) or a component thereof (e.g., ameliorationmodule 230, FIG. 2D) alters and/or coordinates the alteration of theencoding format of at least a portion of the non-volatile memory of thestorage device (e.g., storage medium 130 of storage device 120, FIG.1A). In some embodiments, the one or more parameters for theamelioration process include a level of urgency for the ameliorationprocess, a target reduced declared capacity of the non-volatile memoryof the storage device, and/or a target amount of reduction inutilization of the non-volatile memory of the storage device, or anycombination or subset thereof. In some embodiments, altering theencoding format of at least a portion of the non-volatile memory of thestorage device in accordance with one or more parameters for theamelioration process includes altering the encoding format only enoughto meet a target amount of declared capacity reduction specified by theone or more parameters. Stated another way, the one or more parametersof the amelioration process indicate or correspond to a target amount ofreduction in storage capacity available to the host. For example, asdescribed in greater detail below, the encoding format of a portion ofthe non-volatile memory is altered from TLC to MLC if a target amount ofdeclared capacity reduction can be met by doing so, rather than alteringthe encoding format from TLC to SLC. In some embodiments, the one ormore parameters for the amelioration process take into account, orenable the amelioration process to take into account, a projectedreduction in the declared capacity of the non-volatile memory of thestorage device that will be needed in a predefined time period (e.g., anhour, a day, a week, or any other predefined period). For example, insome embodiments, the parameters of the amelioration process indicate aspecific reduction in declared capacity and a reduction in utilization.A host might use the trim operation to reduce utilization and thendirect the storage device to reduce its declared capacity. In accordancewith the decreased capacity and utilization, the storage device altersthe encoding format of a selected portion of the non-volatile storagemedia (as noted above altering may itself be a multi-step process). Insome embodiments, the amelioration process steps of reducingutilization, reducing declared capacity and altering the encoding formatmay occur in any order and may be partially or completely overlapping(e.g., performed in parallel). In some embodiments, the ameliorationprocess may change the declared capacity of the storage device prior tothe completion of the reduction in utilization, wherein the storagedevice operates with reduced over-provisioning until the utilizationreduction is completed.

In some embodiments, altering (704) the encoding format of at least aportion of the non-volatile memory of the storage device includesaltering (720) the encoding format from a higher-density physicalencoding format to a lower-density physical encoding format. Forexample, the encoding format of selectable portion 131 of storage medium130 (FIG. 1A) is altered from MLC to SLC (e.g., a higher-densityphysical encoding format to a lower-density physical encoding format)such that the memory cells comprising selectable portion 131 store fewerbits per cell than under the previous encoding format.

In some embodiments, altering (704) the encoding format of at least aportion of the non-volatile memory of the storage device includesaltering (722) a number of states per memory cell from a higher numberof states to a lower number of states. As described above with respectto FIGS. 5A and 5B, states of a memory cell represent sequential voltageranges within the total voltage range of a memory cell that representdifferent bit-tuples (e.g., in FIG. 5B, sequential voltage ranges 311,312, 313, 314 represent corresponding bit-tuples “11,” “01,” “00,” and“10.”). The number of bits of information in a memory cell of anon-volatile memory is a function of the possible states that can bestored per memory cell, which is determined by the physical encodingformat of the memory cell. Specifically, the number of bits ofinformation that be stored per cell is equal to log₂ N, where N is thecorresponding maximum number of possible states per cell. Therefore, insome embodiments, altering a number of states per memory cell includesaltering the number of detectable voltage levels for a memory cell,thereby altering the amount of information stored in a memory cell.

In some embodiments, altering (704) the encoding format of at least aportion of the non-volatile memory of the storage device includesaltering (724) the encoding format from a Triple-Level Cell (TLC) formatto a Multi-Level Cell (MLC) format.

In some embodiments, altering (704) the encoding format of at least aportion of the non-volatile memory of the storage device includesaltering (726) the encoding format from a Triple-Level Cell (TLC) formatto a Single-Level Cell (SLC) format.

In some embodiments, altering (704) the encoding format of at least aportion of the non-volatile memory of the storage device includesaltering (728) the encoding format from a Multi-Level Cell (MLC) formatto a Single-Level Cell (SLC) format.

In some embodiments, performing (704) the amelioration process to reducedeclared capacity of the non-volatile memory of the storage deviceincludes reducing (730) utilization of the non-volatile memory of thestorage device. As described above with respect to FIG. 2D, in someembodiments, the amelioration process (606) includes a utilizationreduction process (608) (e.g., performed by utilization module 232,FIGS. 2A-1 and 2A-2, utilization module 262, FIGS. 2B-1 and 2B-2, orutilization module 292, FIGS. 2C-1 and 2C-2). Examples of ways in whichutilization of the non-volatile memory of the storage device may bereduced include trimming or unmapping not live data, deletingdiscardable data (e.g., temporary data files) used by a host, and/ormoving data from the storage device to one or more other storagedevices.

In some embodiments, altering (704) the encoding format of at least aportion of the non-volatile memory of the storage device includesaltering (732) the encoding format of a memory portion of a plurality ofmemory portions of the non-volatile memory of the storage device. Forexample, referring to FIG. 4, the encoding format of memory cellscomprising bank 1 (e.g., Bank 420-1 of the plurality of banks, Banks420-1 through 420-q), specifically, is altered from MLC to SLC,resulting in the memory cells of bank 1 storing fewer bits per cell thanthe memory cells of other banks of the non-volatile memory.

In some embodiments, altering (704) the encoding format of at least aportion of the non-volatile memory of the storage device includesaltering (734) the encoding format of all client data of thenon-volatile memory of the storage device. For example, the encodingformat of all memory cells comprising storage medium 130 (FIG. 1A) isaltered from MLC to SLC, resulting in those memory cells of storagemedium 130 storing one bit of information per cell, rather than twobits. In some embodiments, a storage device may store all client data ina high-density encoding format (e.g., TLC or MLC) but store storagedevice metadata (e.g., mapping table 402) in a different encoding format(e.g., SLC). The amelioration process may result in all of the storagemedia associated with the client data being converted to a lower-densityencoding format while the storage device metadata remains in itsoriginal encoding format. In some embodiments, the retention of theclient data is optional. For example, the amelioration process may beperformed much faster if the client data is not retained, similar to aSCSI low-level format operation.

In some embodiments, performing the amelioration process (704) to reducedeclared capacity of the non-volatile memory of the storage devicefurther includes advertising (736) a reduced declared capacity of thenon-volatile memory of the storage device. In some embodiments, storagecontroller 124 (FIG. 1A) or a component thereof (e.g., advertisingmodule 239 of capacity module 234, FIG. 2D) advertises/or coordinatesthe advertisement of a reduced declared capacity of non-volatile memoryof a storage device (e.g., storage medium 130 of storage device 120,FIG. 1A) or storage subsystem. In some implementations, the storagedevice, or a corresponding storage controller, cluster controller,management module or data storage system sends a message to the hostadvertising the reduced declared capacity of the non-volatile memory ofthe storage device. In some implementations, advertising the reduceddeclared capacity is accomplished by sending an interrupt or othermessage to the host.

In some implementations, advertising the reduced declared capacity isaccomplished by receiving a query from a host to which the storagedevice is operatively coupled, and in response to receiving the query,reporting the reduced declared capacity of the non-volatile memory ofthe storage device. In some such implementations, the host is configuredto periodically query the storage system, storage controller, managementmodule, cluster controller or storage device, for example for a systemor device health status.

In some implementations, advertising the reduced declared capacity isaccomplished by receiving a command (e.g., a storage read or writecommand) from a host to which the storage device is operatively coupled,and in response to receiving the command, sending a response to thecommand that includes a notification of the reduced declared capacity ofthe non-volatile memory of the storage device.

In some implementations, advertising the reduced declared capacity isaccomplished by receiving a command (e.g., a storage read or writecommand) from a host to which the storage device is operatively coupled,and in response to receiving the command, sending both a response to thecommand and a notification that prompts the host to obtain information,including the reduced declared capacity of the non-volatile memory ofthe storage device, from the storage device or from the data storagesystem that includes the storage device. In some embodiments, themechanism used for returning a notification when responding to a commandfrom the host is a SCSI deferred error or deferred error response code.

In some embodiments, after beginning performance of the ameliorationprocess (704) to reduce declared capacity of the non-volatile memory ofthe storage device, the method includes detecting an indication (738) toabort the reduction in declared capacity of the non-volatile memory ofthe storage device; and in response to detecting the indication to abortthe reduction in declared capacity of the non-volatile memory of thestorage device, aborting (740) performance of the amelioration processto reduce declared capacity of the non-volatile memory of the storagedevice. In the context of these embodiments, detecting the indication toabort is herein defined to mean either receiving a signal to abort thereduction in declared capacity (e.g., receiving the signal from acontroller of the storage device or a storage system controller of astorage system that includes the storage device) or evaluating one ormore metrics of the storage device and based on the evaluation,determining to abort the reduction in declared capacity. For example,during the operation of the amelioration process, normal storageoperations will continue to be performed (e.g., read, write, delete,trim, etc.). Normal storage operations include operations like trim thatexplicitly reduce the storage device utilization, possibly enough tomerit aborting the amelioration process. Other storage activity such asgarbage collection may also reduce utilization, possibly enough to meritaborting the amelioration process.

In some embodiments the amelioration process (e.g., periodically,semi-continuously, initially, finally, occasionally or irregularly)recomputes or re-evaluates a number of parameters, such as the targetreduced declared capacity and/or the target amount of utilizationreduction), as those parameters may change in value due to theamelioration process and/or normal storage operations (e.g., read,write, erase and trim or unmap operations). In some circumstances, inaccordance with the recomputed or re-evaluated parameters, one or moreportions of the amelioration process, such as the utilization reductionprocess, is re-prioritized, re-scheduled, or aborted.

In some embodiments, the storage device includes (742) one or more flashmemory devices. In some embodiments, the storage device comprises astorage medium (e.g., storage medium 130, FIG. 1A), and the storagemedium comprises one or more non-volatile storage devices, such as flashmemory devices. In some embodiments, the storage medium (e.g., storagemedium 130, FIG. 1A) is a single flash memory device, while in otherembodiments the storage medium includes a plurality of flash memorydevices. For example, in some embodiments, the storage medium includesdozens or hundreds of flash memory devices, organized in parallel memorychannels, such as 16, 32 or 64 flash memory devices per memory channel,and 8, 16 or 32 parallel memory channels. In some embodiments, thenon-volatile storage medium (e.g., storage medium 130, FIG. 1A) includesNAND-type flash memory or NOR-type flash memory. In other embodiments,the storage medium comprises one or more other types of non-volatilestorage devices.

FIGS. 8A-8C illustrate a flowchart representation of a method 800 ofmanaging a storage system, in accordance with some embodiments. In someembodiments, the method 800 includes detecting (802) an ameliorationtrigger for reducing declared capacity of non-volatile memory of astorage device (e.g., storage device 120 of FIG. 1A, or any of storagedevices 160-1 to 160-m of FIG. 1B, or any of storage devices 194-1 to194-n or 194-j to 194-k of FIG. 1C) of the storage system (e.g., datastorage system 100, FIG. 1A; data storage system 140, FIG. 1B; or datastorage system 170, FIG. 1C), and in accordance with the detectedamelioration trigger, performing (804) an amelioration process to reducedeclared capacity of the non-volatile memory of the storage device, theperforming including: deleting from the storage device discardable datathat is used by a host (e.g., computer system 110 of FIG. 1A, computersystem 142 of FIG. 1B, or computer system 172 of FIG. 1C), and reducingdeclared capacity of the non-volatile memory of the storage device. Insome embodiments, reducing declared capacity of non-volatile memory ofthe storage device includes reducing storage capacity, of thenon-volatile memory of the storage device, available to the host. Thedeclared capacity of non-volatile memory of the storage device issometimes called the advertised capacity, and is typically used by theoperating system and/or a file system of the host as the maximumcapacity that the host's operating system or file system will attempt toallocate.

In some embodiments, the amelioration trigger is detected (802) by anamelioration module (e.g., amelioration module 230 of FIG. 2A-1, 2A-2 or2D, 260 of FIG. 2B-1 or 2B-2, or 290 of FIG. 2C-1 or 2C-2), or acomponent thereof (e.g., detection module 231 of FIG. 2A-1, 2A-2 or 2D,261 of FIG. 2B-1 or 2B-2, or 291 of FIG. 2C-1 or 2C-2). In someembodiments, detecting the amelioration trigger includes receiving orgenerating the amelioration trigger. Furthermore, in some embodiments,prior to detecting the amelioration trigger (802), method 800 includesdetecting a wear condition and reducing over-provisioning of thenon-volatile memory of the storage device, without reducing declaredcapacity of the non-volatile memory of the storage device, as describedabove with respect to operation 601 of FIG. 6.

In some embodiments, an amelioration module (e.g., amelioration module230, FIGS. 2A-1 and 2A-2, amelioration module 260, FIGS. 2B-1 and 2B-2,and/or amelioration module 290, FIGS. 2C-1 and 2C-2) is used to perform(804) the amelioration process, in accordance with the ameliorationtrigger, to reduce declared capacity of the non-volatile memory of thestorage device. In some embodiments, the amelioration module includes autilization module (e.g., specifically, deleting module 236 ofutilization module 232, FIG. 2D) for deleting from the storage devicediscardable data that is used by a host, and a capacity module (e.g.,capacity module 234) for reducing declared capacity of the non-volatilememory of the storage device.

In some embodiments, the host (e.g., computer system 110 of FIG. 1A,computer system 142 of FIG. 1B, or computer system 172 of FIG. 1C)includes (806) a client on behalf of which data is stored in the storagesystem (e.g., data storage system 100, FIG. 1A; data storage system 140,FIG. 1B; data storage system 170, FIG. 1C). In some embodiments, theclient is or includes an entity on behalf of which data is stored in thestorage system.

In some embodiments, the host includes (808) a storage system controller(e.g., storage system controller 150, FIG. 1B) of the storage system(e.g., data storage system 140, FIG. 1B). In some of these embodiments,the storage system controller controls and/or coordinates operationsamong one or more storage devices. In some of these embodiments, thedata storage system (e.g., data storage system 140, FIG. 1B) is called ascale-up system.

In some embodiments, the host includes (810) a cluster controller (e.g.,cluster controller 180, FIG. 1C) of the storage system (e.g., datastorage system 170, FIG. 1C). In some of these embodiments, the clustercontroller controls and/or coordinates operations among one or more datastorage subsystems, as shown for example in FIG. 1C, where each of thedata storage subsystems may be implemented as a data storage systemhaving one or more storage devices (e.g., data storage system 140, FIG.1B). In some of these embodiments, the data storage system (e.g., datastorage system 170, FIG. 1C) is called a scale-out system or a clusteredstorage system.

In some embodiments, the detecting, the performing, or both thedetecting and the performing are performed (812) by the storage device(e.g., storage device 120, FIG. 1A, storage device 160, FIG. 1B, orstorage device 194, FIG. 1C) or one or more components of the storagedevice (e.g., storage controller 124, FIG. 1A). In some embodiments,method 800, or at least the detecting operation 802 and/or performingoperation 804 of method 800, is governed by instructions that are storedin a non-transitory computer readable storage medium and that areexecuted by one or more processors of a device, such as the one or moreprocessing units (CPUs) 122-1 of management module 121-1, shown in FIGS.1A and 2A-1.

In some embodiments, the detecting, the performing, or both thedetecting and the performing are performed (814) by one or moresubsystems of the storage system distinct from the storage device. Forexample, in some of these embodiments, the detecting, the performing, orboth the detecting and the performing are performed by a storage systemcontroller (e.g., storage system controller 150, FIG. 1B) of the storagesystem (e.g., data storage system 140, FIG. 1B). In some embodiments,method 800, or at least the detecting operation 802 and/or performingoperation 804 of method 800, is governed by instructions that are storedin a non-transitory computer readable storage medium and that areexecuted by one or more processors of a device, such as the one or moreprocessing units (CPUs) 152-1 of management module 151-1, shown in FIGS.1B and 2B-1.

In some embodiments, the detecting, the performing, or both thedetecting and the performing are performed (816), at least in part, bythe host. In some embodiments, method 800, or at least the detectingoperation 802 and/or performing operation 804 of method 800, is governedat least in part by instructions that are stored in a non-transitorycomputer readable storage medium and that are executed by one or moreprocessors of a host (processors not shown in FIGS. 1A, 1B and 1C), suchas the one or more processing units (CPUs) of management module 121-2(FIGS. 1A and 2A-2); or management module 151-2 (FIGS. 1B and 2B-2); ormanagement module 181-2 (FIGS. 1C and 2C-2).

In some embodiments, deleting (804) discardable data that is used by thehost includes deleting (818) discardable data that is used by the hostin accordance with one or more parameters for the amelioration process.In some embodiments, storage controller 124 (FIG. 1A) or a componentthereof (e.g., deleting module 236 of utilization module 232, FIG. 2D)deletes and/or coordinates the deletion of, from a storage device (e.g.,storage medium 130 of storage device 120, FIG. 1A), discardable datathat is used by the host. In some embodiments, the one or moreparameters for the amelioration process include a level of urgency forthe amelioration process, a target reduced declared capacity of thenon-volatile memory of the storage device, and/or a target amount ofreduction in utilization of the non-volatile memory of the storagedevice, or any combination or subset thereof. In some embodiments,deleting discardable data (e.g., temporary data) that is used by thehost in accordance with one or more parameters for the ameliorationprocess includes deleting only discardable data to meet a target amountof declared capacity reduction specified by the one or more parameters.Stated another way, the one or more parameters of the ameliorationprocess indicate or correspond to a target amount of reduction instorage capacity available to the host. In some embodiments, the one ormore parameters for the amelioration process take into account, orenable the amelioration process to take into account, a projectedreduction in the declared capacity of the non-volatile memory of thestorage device that will be needed in a predefined time period (e.g., anhour, a day, a week, or any other predefined time period).

In some embodiments, deleting (804) discardable data that is used by thehost includes deleting (820) temporary data. In some embodiments,temporary data is stored in a storage device (e.g., storage medium 130of storage device 120, FIG. 1A), and includes, for example, temporarydata of an Operating System installed on the host (e.g., computer system110, FIG. 1A), or temporary data generated by one or more applications,system processes, and/or programs used by the host. For example, in theUnix or Linux operating systems, files stored in the directory “/tmp”are known to be discardable if they have not been used within aspecified period of time (typically an hour or a day, etc.).

In some embodiments, deleting (804) discardable data that is used by thehost includes deleting (822) data that can be rebuilt when needed. Datathat can be rebuilt includes, for example, certain indexes, caches, andother file types that are generated with reference to data stored in thememory of a host (e.g., computer system 110, FIG. 1A) and/or storagedevice (e.g., storage device 120 of FIG. 1A, or any of storage devices160-1 to 160-m of FIG. 1B, or any of storage devices 194-1 to 194-n or194-j to 194-k of FIG. 1C) of a storage system and can be regenerated ifneeded.

In some embodiments, deleting (804) discardable data that is used by thehost includes deleting (824) snapshots of data. Snapshots of datainclude, for example, back-up files/images (e.g., for recovering data),virtual machine snapshots (e.g., for capturing the state, data, andhardware configuration of a virtual machine), and other types of filesused for capturing the state of data stored in the memory of a host(e.g., computer system 110, FIG. 1A) and/or storage device (e.g.,storage device 120 of FIG. 1A, or any of storage devices 160-1 to 160-mof FIG. 1B, or any of storage devices 194-1 to 194-n or 194-j to 194-kof FIG. 1C) of a storage system.

In some embodiments, deleting (804) discardable data that is used by thehost includes deleting (826) data that is pre-marked by the host as datathat is discardable. In some embodiments, such data includes preloadeddata that the user did not request, one example of which is factorypreloaded data stored in storage device 120 by the manufacturer (e.g.,programs, applications, back-up files for restoring the OS, etc.).Another example of preloaded data that the user did not request is datapreloaded by a website or online application in anticipation of futurerequests by a user.

In some embodiments, deleting (804) discardable data that is used by thehost includes invalidating (828) one or more logical address entries ofa mapping table (e.g., by using a trim operation), where the logicaladdress entries are associated with the discardable data, and themapping table is used to translate logical addresses in a logicaladdress space to physical addresses in a physical address space of thestorage device. For example, with reference to FIG. 4, deletingdiscardable data includes invalidating logical address entry LBA 0 ofmapping table 402, where LBA 0 is mapped to bank 1 (e.g., Bank 420-1),block 3 (e.g., Block 421-3), page 37 (pages not shown in FIG. 4) ofphysical address space 410.

In some embodiments, reducing (804) declared capacity of thenon-volatile memory of the storage device includes making (830) a numberof logical addresses, less than or equal to a number of logicaladdresses corresponding to the invalidated logical address entries,unavailable to the host. In some embodiments, storage controller 124(FIG. 1A) or a component thereof (e.g., LBA reduction module 238 ofcapacity module 234, FIG. 2D) reduces and/or coordinates reduction oflogical addresses of the logical block address space (e.g., LBA space320, FIG. 3). In some embodiments, each of the logical address entriescorresponds to multiple (e.g., eight) logical addresses (e.g., 8 LBAsper logical address entry).

In some embodiments, performing the amelioration process (804) to reducedeclared capacity of the non-volatile memory of the storage devicefurther includes advertising (832) a reduced declared capacity of thenon-volatile memory of the storage device. In some embodiments, storagecontroller 124 (FIG. 1A) or a component thereof (e.g., advertisingmodule 239 of capacity module 234, FIG. 2D) advertises/or coordinatesthe advertisement of a reduced declared capacity of non-volatile memoryof a storage device (e.g., storage medium 130 of storage device 120,FIG. 1A) or storage subsystem. In some implementations, the storagedevice, or a corresponding storage controller, cluster controller,management module or data storage system sends a message to the hostadvertising the reduced declared capacity of the non-volatile memory ofthe storage device. In some implementations, advertising the reduceddeclared capacity is accomplished by sending an interrupt or otherin-band or out-of-band message to the host.

In some implementations, advertising the reduced declared capacity isaccomplished by receiving a query from a host to which the storagedevice is operatively coupled, and in response to receiving the query,reporting the reduced declared capacity of the non-volatile memory ofthe storage device. In some such implementations, the host is configuredto periodically query the storage system, storage controller, managementmodule, cluster controller or storage device, for example for a systemor device health status.

In some implementations, advertising the reduced declared capacity isaccomplished by receiving a command (e.g., a storage read or writecommand) from a host to which the storage device is operatively coupled,and in response to receiving the command, sending a response to thecommand that includes a notification of the reduced declared capacity ofthe non-volatile memory of the storage device.

In some implementations, advertising the reduced declared capacity isaccomplished by receiving a command (e.g., a storage read or writecommand) from a host to which the storage device is operatively coupled,and in response to receiving the command, sending both a response to thecommand and a notification that prompts the host to obtain information,including the reduced declared capacity of the non-volatile memory ofthe storage device, from the storage device or from the data storagesystem that includes the storage device. In some embodiments, themechanism used for returning a notification when responding to a commandfrom the host is a SCSI deferred error or deferred error response code.

In some embodiments, after beginning performance of the ameliorationprocess (804) to reduce declared capacity of the non-volatile memory ofthe storage device, the method includes detecting an indication (834) toabort the reduction in declared capacity of the non-volatile memory ofthe storage device; and in response to detecting the indication to abortthe reduction in declared capacity of the non-volatile memory of thestorage device, aborting (836) performance of the amelioration processto reduce declared capacity of the non-volatile memory of the storagedevice. Detecting the indication to abort is herein defined to meaneither receiving a signal to abort the reduction in declared capacity(e.g., receiving the signal from a controller of the storage device or astorage system controller of a storage system that includes the storagedevice) or evaluating one or more metrics of the storage device andbased on the evaluation, determining to abort the reduction in declaredcapacity. For example, during the operation of the amelioration process,normal storage operations will continue to be performed (e.g., read,write, trim, etc.). Normal storage operations include operations liketrim that explicitly reduce the storage device utilization, possiblyenough to merit aborting the amelioration process. Other storageactivity such as garbage collection may also reduce utilization,possibly enough to merit aborting the amelioration process.

In some embodiments the amelioration process (e.g., periodically,semi-continuously, initially, finally, occasionally or irregularly)recomputes or re-evaluates a number of parameters, such as the targetreduced declared capacity and/or the target amount of utilizationreduction), as those parameters may change in value due to theamelioration process and/or normal storage operations (e.g., read,write, erase and trim or unmap operations). In some circumstances, inaccordance with the recomputed or re-evaluated parameters, one or moreportions of the amelioration process, such as the utilization reductionprocess, is re-prioritized, re-scheduled, or aborted.

In some embodiments, the storage device includes (838) one or more flashmemory devices. In some embodiments, the storage device comprises astorage medium (e.g., storage medium 130, FIG. 1A), and the storagemedium comprises one or more non-volatile storage devices, such as flashmemory devices. In some embodiments, the storage medium (e.g., storagemedium 130, FIG. 1A) is a single flash memory device, while in otherembodiments the storage medium includes a plurality of flash memorydevices. For example, in some embodiments, the storage medium includesdozens or hundreds of flash memory devices, organized in parallel memorychannels, such as 16, 32 or 64 flash memory devices per memory channel,and 8, 16 or 32 parallel memory channels. In some embodiments, thenon-volatile storage medium (e.g., storage medium 130, FIG. 1A) includesNAND-type flash memory or NOR-type flash memory. In other embodiments,the storage medium comprises one or more other types of non-volatilestorage devices.

FIGS. 9A-9C illustrate a flowchart representation of a method 900 ofmanaging a storage system, in accordance with some embodiments. In someembodiments, the method 900 includes detecting (902) an ameliorationtrigger for reducing declared capacity of non-volatile memory of a firststorage device (e.g., storage device 120 of FIG. 1A, or any of storagedevices 160-1 to 160-m of FIG. 1B, or any of storage devices 194-1 to194-n or 194-j to 194-k of FIG. 1C) of the storage system (e.g., datastorage system 100, FIG. 1A; data storage system 140, FIG. 1B; or datastorage system 170, FIG. 1C), and in accordance with the detectedamelioration trigger, performing (904) an amelioration process to reducedeclared capacity of the non-volatile memory of the first storagedevice, the performing including: moving a portion of data (e.g., livedata 334, FIG. 3) that is used by a host (e.g., computer system 110 ofFIG. 1A, computer system 142 of FIG. 1B, or computer system 172 of FIG.1C) from the first storage device to another storage device of thestorage system, and reducing declared capacity of the non-volatilememory of the first storage device. In some embodiments, detecting anamelioration trigger for reducing declared capacity of non-volatilememory of a first storage device of the storage system includesgenerating the amelioration trigger for reducing declared capacity ofnon-volatile memory of the first storage device of the storage system.In some embodiments, reducing declared capacity of non-volatile memoryof the first storage device includes reducing storage capacity, of thenon-volatile memory of the first storage device, available to the host.The declared capacity of non-volatile memory of the first storage deviceis sometimes called the advertised capacity, and is typically used bythe operating system and/or a file system of the host as the maximumcapacity that the host's operating system or file system will attempt toallocate.

In some embodiments, the amelioration trigger is detected (902) by anamelioration module (e.g., amelioration module 230 of FIG. 2A-1, 2A-2 or2D, 260 of FIG. 2B-1 or 2B-2, or 290 of FIG. 2C-1 or 2C-2), or acomponent thereof (e.g., detection module 231 of FIG. 2A-1, 2A-2 or 2D,261 of FIG. 2B-1 or 2B-2, or 291 of FIG. 2C-1 or 2C-2). In someembodiments, detecting the amelioration trigger includes receiving orgenerating the amelioration trigger. Furthermore, in some embodiments,prior to detecting the amelioration trigger (902), method 900 includesdetecting a wear condition and reducing over-provisioning of thenon-volatile memory of the first storage device, without reducingdeclared capacity of the non-volatile memory of the first storagedevice, as described above with respect to operation 601 of FIG. 6.

In some embodiments, an amelioration module (e.g., amelioration module230, FIGS. 2A-1 and 2A-2, amelioration module 260, FIGS. 2B-1 and 2B-2,and/or amelioration module 290, FIGS. 2C-1 and 2C-2) is used to perform(904) the amelioration process, in accordance with the ameliorationtrigger, to reduce declared capacity of the non-volatile memory of thefirst storage device. In some embodiments, the amelioration moduleincludes a utilization module (e.g., specifically, moving module 237 ofutilization module 232, FIG. 2D) for moving a portion of data that isused by a host from the first storage device to another storage deviceof the storage system, and a capacity module (e.g., capacity module 234)for reducing declared capacity of the non-volatile memory of the firststorage device.

In some embodiments, the host (e.g., computer system 110 of FIG. 1A,computer system 142 of FIG. 1B, or computer system 172 of FIG. 1C)includes (906) a client on behalf of which data is stored in the storagesystem (e.g., data storage system 100, FIG. 1A; data storage system 140,FIG. 1B; data storage system 170, FIG. 1C). In some embodiments, theclient is or includes an entity on behalf of which data is stored in thestorage system.

In some embodiments, the host includes (908) a storage system controller(e.g., storage system controller 150, FIG. 1B) of the storage system(e.g., data storage system 140, FIG. 1B). In some of these embodiments,the storage system controller controls and/or coordinates operationsamong one or more storage devices. In some of these embodiments, thedata storage system (e.g., data storage system 140, FIG. 1B) is called ascale-up system.

In some embodiments, the host includes (910) a cluster controller (e.g.,cluster controller 180, FIG. 1C) of the storage system (e.g., datastorage system 170, FIG. 1C). In some of these embodiments, the clustercontroller controls and/or coordinates operations among one or more datastorage subsystems, as shown for example in FIG. 1C, where each of thedata storage subsystems may be implemented as a data storage systemhaving one or more storage devices (e.g., data storage system 140, FIG.1B). In some of these embodiments, the data storage system (e.g., datastorage system 170, FIG. 1C) is called a scale-out system or a clusteredstorage system.

In some embodiments, the detecting, the performing, or both thedetecting and the performing are performed (912), at least in part, bythe host. In some embodiments, method 900, or at least the detectingoperation 902 and/or performing operation 904 of method 900, is governedat least in part by instructions that are stored in a non-transitorycomputer readable storage medium and that are executed by one or moreprocessors of a host (processors not shown in FIGS. 1A, 1B and 1C), suchas the one or more processing units (CPUs) of management module 121-2(FIGS. 1A and 2A-2); or management module 151-2 (FIGS. 1B and 2B-2); ormanagement module 181-2 (FIGS. 1C and 2C-2).

In some embodiments, the detecting, the performing, or both thedetecting and the performing are performed (914) by one or moresubsystems of the storage system distinct from the first storage device.For example, in some of these embodiments, the detecting, theperforming, or both the detecting and the performing are performed by astorage system controller (e.g., storage system controller 150, FIG. 1B)of the storage system (e.g., data storage system 140, FIG. 1B). In someembodiments, method 900, or at least the detecting operation 902 and/orperforming operation 904 of method 900, is governed by instructions thatare stored in a non-transitory computer readable storage medium and thatare executed by one or more processors of a device, such as the one ormore processing units (CPUs) 152-1 of management module 151-1, shown inFIGS. 1B and 2B-1.

In some embodiments, moving (904) the portion of data that is used bythe host includes moving (916) the portion of data that is used by thehost in accordance with one or more parameters for the ameliorationprocess. In some embodiments, storage controller 124 (FIG. 1A) or acomponent thereof (e.g., moving module 237 of utilization module 232,FIG. 2D) moves and/or coordinates the moving of data that is used by thehost from a first storage device (e.g., storage medium 130 of storagedevice 120, FIG. 1A), or a physical location within the first storagedevice where the data is stored, to another storage device. In someembodiments, the one or more parameters for the amelioration processinclude a level of urgency for the amelioration process, a targetreduced declared capacity of the non-volatile memory of the firststorage device, and/or a target amount of reduction in utilization ofthe non-volatile memory of the first storage device, or any combinationor subset thereof. In some embodiments, moving the portion of data thatis used by the host in accordance with one or more parameters for theamelioration process includes moving enough data to meet a target amountof declared capacity reduction specified by the one or more parameters.Stated another way, the one or more parameters of the ameliorationprocess indicate or correspond to a target amount of reduction instorage capacity available to the host. In some embodiments, the one ormore parameters for the amelioration process take into account, orenable the amelioration process to take into account, a projectedreduction in the declared capacity of the non-volatile memory of thefirst storage device that will be needed in a predefined time period(e.g., an hour, a day, a week, or any other predefined time period).

In some embodiments, moving (904) the portion of data that is used bythe host includes (922): moving data associated with one or more virtuallogical addresses, including updating a virtual address mapping module.Referring to FIG. 2B-1, for example, system mapping module 250-1 is anexample of a virtual address mapping module that maps virtual logicaladdresses to intermediate addresses, each of which identifies a storagedevice and a location within the storage device at which data,corresponding to a virtual logical address, is stored. Similarly,cluster mapping module 280-1, FIG. 2C-1, is an example of a virtualaddress mapping module that maps virtual logical addresses tointermediate addresses, each of which identifies a storage device and alocation within the storage device at which data, corresponding to avirtual logical address, is stored.

In some embodiments, moving the portion of the data includes selecting(918) one or more logical addresses of that data so as to minimizeperformance degradation. Stated another way, in some embodiments, themethod includes selecting which logical address entries to move so as tominimize performance degradation. For example, to achieve a particularamount of declared capacity reduction, in some circumstances there maybe multiple candidates, or sets of candidates, of logical addresses forwhich a mapping in a virtual address mapping module (e.g., systemmapping module 250, FIG. 2B-1 or FIG. 2B-2, or cluster mapping module280, FIG. 2C-1 or FIG. 2C-2) could be altered (e.g., by movingassociated data, and subsequently updating the virtual address mappingmodule) to help accomplish the reduction in declared storage capacity ofthe non-volatile memory of the first storage device. Further, some ofthe sets of candidates may be accessed or overwritten less frequentlythan other sets of candidates. By selecting (918) the one or morelogical addresses from among the sets of candidates that are accessed oroverwritten the least frequently, the process minimizes performancedegradation. In another example, the logical addresses are selected inaccordance with a wear-leveling methodology, so as to promote uniformwearing, or to avoid uneven wearing of the non-volatile memory media.

In some embodiments, prior to moving data associated with one or morelogical addresses, the one or more logical addresses are selected (920)so as to minimize overhead from garbage collection. While updating thevirtual address mapping module may, at least in some circumstances, beaccomplished without physically moving data from one location to anotherwithin the physical address space of the non-volatile memory of thefirst storage device, in practice the underlying cause of the detectedamelioration trigger may dictate the physical movement of some data fromthe first storage device to another storage device. In accordance withsome embodiments, the device or system that does the selecting (920)uses a process or analytical method to minimize the amount of datamovement that will result from altering the virtual address mappingmodule.

In some embodiments, reducing (904) declared capacity of thenon-volatile memory of the first storage device includes trimming (924)logical addresses associated with the data moved from the first storagedevice. As a result, the “old copy” of the moved data is no longeraccessible.

In some embodiments, storage controller 124 (FIG. 1A) or a componentthereof (e.g., LBA reduction module 238 of capacity module 234, FIG. 2D)reduces and/or coordinates reduction of logical addresses of the logicalblock address space (e.g., LBA space 320, FIG. 3). In some embodiments,the logical addresses are those logical addresses associated with thefirst storage device prior to performing the amelioration process. Insome embodiments, each of the logical address entries corresponds tomultiple (e.g., eight) logical addresses (e.g., 8 LBAs per logicaladdress entry).

In some embodiments, performing (904) the amelioration process to reducedeclared capacity of the non-volatile memory of the first storage devicefurther includes invalidating (926) one or more logical address entries,associated with the portion of data, of a mapping table, the mappingtable used to translate logical addresses in the logical address spaceto physical addresses in a physical address space of the first storagedevice. For example, with reference to FIG. 4, performing theamelioration process includes invalidating logical address entry LBA 0of mapping table 402, where LBA 0 is mapped to bank 1 (e.g., Bank420-1), block 3 (e.g., Block 421-3), page 37 (pages not shown in FIG. 4)of physical address space 410. Corresponding memory blocks, to which theinvalidated data and a number of logical addresses correspond, aretherefore prepared for being made unavailable to the host, for deletion,and/or for garbage collection (e.g., valid pages of a block arere-written to a new block, and the old block containing invalid data iserased such that the old block is available for new data to be written).In some embodiments, performing the amelioration process includessetting one or more indicators for one or more physical addressescorresponding to the one or more invalidated logical address entries toindicate that data stored at the one or more physical addresses can bediscarded. Alternatively, a record or other data corresponding to one ormore of the physical addresses corresponding to the one or moreinvalidated logical address entries is moved to a list of memoryportions that are eligible or ready for deletion.

In some embodiments, performing the amelioration process (904) to reducedeclared capacity of the non-volatile memory of the first storage devicefurther includes advertising (928) a reduced declared capacity of thenon-volatile memory of the first storage device. In some embodiments,storage controller 124 (FIG. 1A) or a component thereof (e.g.,advertising module 239 of capacity module 234, FIG. 2D) advertises/orcoordinates the advertisement of a reduced declared capacity ofnon-volatile memory of a first storage device (e.g., storage medium 130of storage device 120, FIG. 1A) or storage subsystem. In someimplementations, the first storage device, or a corresponding storagecontroller, cluster controller, management module or data storage systemsends a message to the host advertising the reduced declared capacity ofthe non-volatile memory of the first storage device. In someimplementations, advertising the reduced declared capacity isaccomplished by sending an interrupt or other in-band or out-of-bandmessage to the host.

In some implementations, advertising the reduced declared capacity isaccomplished by receiving a query from a host to which the first storagedevice is operatively coupled, and in response to receiving the query,reporting the reduced declared capacity of the non-volatile memory ofthe first storage device. In some such implementations, the host isconfigured to periodically query the storage system, storage controller,management module, cluster controller or storage device, for example fora system or device health status.

In some implementations, advertising the reduced declared capacity isaccomplished by receiving a command (e.g., a storage read or writecommand) from a host to which the first storage device is operativelycoupled, and in response to receiving the command, sending a response tothe command that includes a notification of the reduced declaredcapacity of the non-volatile memory of the first storage device.

In some implementations, advertising the reduced declared capacity isaccomplished by receiving a command (e.g., a storage read or writecommand) from a host to which the first storage device is operativelycoupled, and in response to receiving the command, sending a response tothe command that includes a notification of the reduced declaredcapacity of the non-volatile memory of the first storage device.

In some implementations, advertising the reduced declared capacity isaccomplished by receiving a command (e.g., a storage read or writecommand) from a host to which the first storage device is operativelycoupled, and in response to receiving the command, sending both aresponse to the command and a notification that prompts the host toobtain information, including the reduced declared capacity of thenon-volatile memory of the first storage device, from the first storagedevice or from the data storage system that includes the first storagedevice. In some embodiments, the mechanism used for returning anotification when responding to a command from the host is a SCSIdeferred error or deferred error response code.

In some embodiments, after beginning performance of the ameliorationprocess (904) to reduce declared capacity of the non-volatile memory ofthe first storage device, the method includes detecting an indication(930) to abort the reduction in declared capacity of the non-volatilememory of the first storage device; and in response to detecting theindication to abort the reduction in declared capacity of thenon-volatile memory of the first storage device, aborting (932)performance of the amelioration process to reduce declared capacity ofthe non-volatile memory of the first storage device. Detecting theindication to abort is herein defined to mean either receiving a signalto abort the reduction in declared capacity (e.g., receiving the signalfrom a controller of the first storage device or a storage systemcontroller of a storage system that includes the first storage device)or evaluating one or more metrics of the first storage device and basedon the evaluation, determining to abort the reduction in declaredcapacity. For example, during the operation of the amelioration process,normal storage operations will continue to be performed (e.g., read,write, delete, trim, etc.). Normal storage operations include operationslike trim that explicitly reduce the first storage device utilization,possibly enough to merit aborting the amelioration process. Otherstorage activity such as garbage collection may also reduce utilization,possibly enough to merit aborting the amelioration process.

In some embodiments the amelioration process (e.g., periodically,semi-continuously, initially, finally, occasionally or irregularly)recomputes or re-evaluates a number of parameters, such as the targetreduced declared capacity and/or the target amount of utilizationreduction), as those parameters may change in value due to theamelioration process and/or normal storage operations (e.g., read,write, erase and trim or unmap operations). In some circumstances, inaccordance with the recomputed or re-evaluated parameters, one or moreportions of the amelioration process, such as the utilization reductionprocess, is re-prioritized, re-scheduled, or aborted.

In some embodiments, the first storage device includes (934) one or moreflash memory devices. In some embodiments, the first storage devicecomprises a storage medium (e.g., storage medium 130, FIG. 1A), and thestorage medium comprises one or more non-volatile storage devices, suchas flash memory devices. In some embodiments, the storage medium (e.g.,storage medium 130, FIG. 1A) is a single flash memory device, while inother embodiments the storage medium includes a plurality of flashmemory devices. For example, in some embodiments, the storage mediumincludes dozens or hundreds of flash memory devices, organized inparallel memory channels, such as 16, 32 or 64 flash memory devices permemory channel, and 8, 16 or 32 parallel memory channels. In someembodiments, the non-volatile storage medium (e.g., storage medium 130,FIG. 1A) includes NAND-type flash memory or NOR-type flash memory. Inother embodiments, the storage medium comprises one or more other typesof non-volatile storage devices.

In embodiments with a plurality of storage devices, a storage system mayfind it advantageous to construct a virtual logical address space (LBASpace 320) out of the logical address spaces, or portions of the logicaladdress spaces, for (or associated with) a plurality of storage devices.The storage system (e.g., Data Storage System 140, Data Storage System170, etc.) includes a virtual mapping module (e.g., system mappingmodule 250-2, cluster mapping module 280-1, etc.) that converts receivedvirtual logical addresses into intermediate addresses. In someembodiments, the virtual mapping module may contain an algorithmicfunction (e.g., CRUSH map of the Ceph storage system). In otherembodiments, the virtual mapping module may be a table of intermediateaddresses indexed by virtual logical address. In some embodiments (e.g.FIG. 1B) the intermediate address consists of one or more references tospecific storage devices (160-x) and one or more logical addresses ofthe one or more referenced storage devices. In some embodiments (e.g.,FIG. 1C), the intermediate address consists of a reference to one ormore storage subsystems (192-x) and one or more locators for data withinthe one or more storage subsystems. A locator is understood by a storagesubsystem to reference one or more logical addresses of one or morestorage devices (194-x) of that storage subsystem. For example, astorage subsystem could use a file system to organize the storagedevices (194-1, 194-2, . . . , etc.) in which case the locator would bea file reference (e.g., file name) or reference to a portion of a file.The usage of the virtual mapping module is advantageous because itallows the storage system flexibility in the location of the dataassociated with the virtual logical address space. In particular, datamay be moved between the plurality of storage devices (or even todifferent locations within a single device) without changing the virtuallogical address by altering the contents of the virtual mapping table.

Semiconductor memory devices include volatile memory devices, such asdynamic random access memory (“DRAM”) or static random access memory(“SRAM”) devices, non-volatile memory devices, such as resistive randomaccess memory (“ReRAM”), electrically erasable programmable read onlymemory (“EEPROM”), flash memory (which can also be considered a subsetof EEPROM), ferroelectric random access memory (“FRAM”), andmagnetoresistive random access memory (“MRAM”), and other semiconductorelements capable of storing information. Each type of memory device mayhave different configurations. For example, flash memory devices may beconfigured in a NAND or a NOR configuration.

The semiconductor memory elements located within and/or over a substratemay be arranged in two or three dimensions, such as a two dimensionalmemory structure or a three dimensional memory structure.

The term “three-dimensional memory device” (or 3D memory device) isherein defined to mean a memory device having multiple memory layers ormultiple levels (e.g., sometimes called multiple memory device levels)of memory elements, including any of the following: a memory devicehaving a monolithic or non-monolithic 3D memory array; or two or more 2Dand/or 3D memory devices, packaged together to form a stacked-chipmemory device.

One of skill in the art will recognize that this invention is notlimited to the two dimensional and three dimensional structuresdescribed but cover all relevant memory structures within the spirit andscope of the invention as described herein and as understood by one ofskill in the art.

It will be understood that, although the terms “first,” “second,” etc.may be used herein to describe various elements, these elements shouldnot be limited by these terms. These terms are only used to distinguishone element from another. For example, a first storage device could betermed a second storage device, and, similarly, a second storage devicecould be termed a first storage device, without changing the meaning ofthe description, so long as all occurrences of the “first storagedevice” are renamed consistently and all occurrences of the “secondstorage device” are renamed consistently. The first storage device andthe second storage device are both storage devices, but they are not thesame storage device.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the claims. Asused in the description of the embodiments and the appended claims, thesingular forms “a,” “an” and “the” are intended to include the pluralforms as well, unless the context clearly indicates otherwise. It willalso be understood that the term “and/or” as used herein refers to andencompasses any and all possible combinations of one or more of theassociated listed items. It will be further understood that the terms“comprises” and/or “comprising,” when used in this specification,specify the presence of stated features, integers, steps, operations,elements, and/or components, but do not preclude the presence oraddition of one or more other features, integers, steps, operations,elements, components, and/or groups thereof.

As used herein, the term “if” may be construed to mean “when” or “upon”or “in response to determining” or “in accordance with a determination”or “in response to detecting,” that a stated condition precedent istrue, depending on the context. Similarly, the phrase “if it isdetermined [that a stated condition precedent is true]” or “if [a statedcondition precedent is true]” or “when [a stated condition precedent istrue]” may be construed to mean “upon determining” or “in response todetermining” or “in accordance with a determination” or “upon detecting”or “in response to detecting” that the stated condition precedent istrue, depending on the context.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the claims to the precise forms disclosed. Many modifications andvariations are possible in view of the above teachings. The embodimentswere chosen and described in order to best explain principles ofoperation and practical applications, to thereby enable others skilledin the art.

What is claimed is:
 1. A method of managing a storage system having aplurality of storage devices, the method comprising: detecting anamelioration trigger for reducing declared capacity of non-volatilememory of a first storage device of the storage system; and inaccordance with the detected amelioration trigger, performing anamelioration process to reduce declared capacity of the non-volatilememory of the first storage device, the performing including: moving aportion of data that is used by a host from the first storage device toanother storage device of the storage system; and reducing declaredcapacity of the non-volatile memory of the first storage device.
 2. Themethod of claim 1, further comprising: prior to detecting theamelioration trigger, detecting a first wear condition of thenon-volatile memory of the first storage device, wherein a total storagecapacity of the non-volatile memory of the first storage device includesdeclared capacity and over-provisioning; and in response to detectingthe first wear condition, performing a remedial action that reducesover-provisioning of the non-volatile memory of the first storage devicewithout reducing declared capacity of the non-volatile memory of thefirst storage device.
 3. The method of claim 2, wherein detecting theamelioration trigger comprises detecting a second wear conditiondistinct from the first wear condition.
 4. The method of claim 1,wherein the host includes a client on behalf of which data is stored inthe storage system.
 5. The method of claim 1, wherein the host includesa storage system controller of the storage system.
 6. The method ofclaim 1, wherein the host includes a cluster controller of the storagesystem.
 7. The method of claim 1, wherein the detecting, the performing,or both the detecting and the performing are performed by the host. 8.The method of claim 1, wherein the detecting, the performing, or boththe detecting and the performing are performed by one or more subsystemsof the storage system distinct from the first storage device.
 9. Themethod of claim 1, wherein moving the portion of data that is used bythe host includes moving the portion of data that is used by the host inaccordance with one or more parameters for the amelioration process. 10.The method of claim 1, wherein moving the portion of data that is usedby the host includes: moving data associated with one or more virtuallogical addresses, including updating a virtual address mapping module.11. The method of claim 1, wherein moving the portion of the dataincludes selecting one or more logical addresses of that data so as tominimize performance degradation.
 12. The method of claim 1, whereinmoving the portion of the data further includes selecting one or morelogical addresses of that data so as to minimize overhead from garbagecollection.
 13. The method of claim 1, wherein performing theamelioration process to reduce declared capacity of the non-volatilememory of the first storage device further includes: invalidating one ormore logical address entries, associated with the portion of data, of amapping table, the mapping table used to translate logical addresses ina logical address space to physical addresses in a physical addressspace of the first storage device.
 14. The method of claim 13, whereinreducing declared capacity of the non-volatile memory of the firststorage device includes trimming logical addresses associated with thedata moved from the first storage device.
 15. The method of claim 1,wherein performing the amelioration process to reduce declared capacityof the non-volatile memory of the first storage device further includesadvertising a reduced declared capacity of the non-volatile memory ofthe first storage device.
 16. The method of claim 1, further comprising:after beginning performance of the amelioration process to reducedeclared capacity of the non-volatile memory of the first storagedevice, detecting an indication to abort the reduction in declaredcapacity of the non-volatile memory of the first storage device; and inresponse to detecting the indication to abort the reduction in declaredcapacity of the non-volatile memory of the first storage device,aborting performance of the amelioration process to reduce declaredcapacity of the non-volatile memory of the first storage device.
 17. Themethod of claim 1, wherein the first storage device comprises one ormore flash memory devices.
 18. A storage system, comprising: a storagemedium; one or more processors; and memory storing one or more programs,which when executed by the one or more processors cause the storagesystem to perform operations comprising: detecting an ameliorationtrigger for reducing declared capacity of non-volatile memory of a firststorage device of the storage system; in accordance with the detectedamelioration trigger, performing an amelioration process to reducedeclared capacity of the non-volatile memory of the first storagedevice, the performing including: moving a portion of data that is usedby a host from the first storage device to another storage device of thestorage system; and reducing declared capacity of the non-volatilememory of the first storage device.
 19. The storage system of claim 18,wherein the one or more programs further include instructions that, whenexecuted by the one or more processors, cause the storage system toperform operations comprising: prior to detecting the ameliorationtrigger, detecting a first wear condition of the non-volatile memory ofthe first storage device, wherein a total storage capacity of thenon-volatile memory of the first storage device includes declaredcapacity and over-provisioning; and in response to detecting the firstwear condition, performing a remedial action that reducesover-provisioning of the non-volatile memory of the first storage devicewithout reducing declared capacity of the non-volatile memory of thefirst storage device.
 20. A non-transitory computer readable storagemedium, storing one or more programs configured for execution by one ormore processors of a storage system, the one or more programs includinginstructions for: detecting an amelioration trigger for reducingdeclared capacity of non-volatile memory of a first storage device ofthe storage system; in accordance with the detected ameliorationtrigger, performing an amelioration process to reduce declared capacityof the non-volatile memory of the first storage device, the performingincluding: moving a portion of data that is used by a host from thefirst storage device to another storage device of the storage system;and reducing declared capacity of the non-volatile memory of the firststorage device.
 21. The non-transitory computer readable storage mediumof claim 20, wherein the one or more programs further includeinstructions for: prior to detecting the amelioration trigger, detectinga first wear condition of the non-volatile memory of the first storagedevice, wherein a total storage capacity of the non-volatile memory ofthe first storage device includes declared capacity andover-provisioning; and in response to detecting the first wearcondition, performing a remedial action that reduces over-provisioningof the non-volatile memory of the first storage device without reducingdeclared capacity of the non-volatile memory of the first storagedevice.